[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 455525818 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002383] x2apic enabled [ 0.003011] Switched APIC routing to physical x2apic. [ 0.004021] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007031] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008015] pid_max: default: 32768 minimum: 301 [ 0.009149] LSM: Security Framework initializing [ 0.010059] Yama: becoming mindful. [ 0.011054] SELinux: Initializing. [ 0.013093] *** VALIDATE selinux *** [ 0.022240] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026772] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027196] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029095] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030117] *** VALIDATE tmpfs *** [ 0.031579] *** VALIDATE proc *** [ 0.033141] *** VALIDATE cgroup *** [ 0.034015] *** VALIDATE cgroup2 *** [ 0.036073] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037156] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040040] Spectre V2 : User space: Vulnerable [ 0.041022] Speculative Store Bypass: Vulnerable [ 0.043000] debug: unmapping init [mem 0xffffffff88c59000-0xffffffff88c60fff] [ 0.045153] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046742] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047030] ... version: 2 [ 0.048016] ... bit width: 48 [ 0.049015] ... generic registers: 4 [ 0.050016] ... value mask: 0000ffffffffffff [ 0.051025] ... max period: 00007fffffffffff [ 0.052017] ... fixed-purpose events: 3 [ 0.052929] ... event mask: 000000070000000f [ 0.053306] rcu: Hierarchical SRCU implementation. [ 0.055493] smp: Bringing up secondary CPUs ... [ 0.056665] x86: Booting SMP configuration: [ 0.057034] .... node #0, CPUs: #1 #2 #3 [ 0.061402] smp: Brought up 1 node, 4 CPUs [ 0.063030] smpboot: Max logical packages: 1 [ 0.064023] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.175814] node 0 deferred pages initialised in 110ms [ 0.179098] devtmpfs: initialized [ 0.180251] x86/mm: Memory block size: 128MB [ 0.183903] gcov: version magic: 0x41383552 [ 0.187297] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.188133] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.190724] pinctrl core: initialized pinctrl subsystem [ 0.193302] [ 0.193924] ************************************************************* [ 0.196017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.198014] ** ** [ 0.200014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.201010] ** ** [ 0.202011] ** This means that this kernel is built to expose internal ** [ 0.203009] ** IOMMU data structures, which may compromise security on ** [ 0.205014] ** your system. ** [ 0.207017] ** ** [ 0.209014] ** If you see this message and you are not debugging the ** [ 0.212018] ** kernel, report this immediately to your vendor! ** [ 0.214014] ** ** [ 0.216016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.218015] ************************************************************* [ 0.221783] NET: Registered protocol family 16 [ 0.223483] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.227082] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.229069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.233146] cpuidle: using governor menu [ 0.234884] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.237569] PCI: Using configuration type 1 for base access [ 0.239137] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.248103] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.251040] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.255064] cryptd: max_cpu_qlen set to 1000 [ 0.258488] ACPI: Added _OSI(Module Device) [ 0.260028] ACPI: Added _OSI(Processor Device) [ 0.261013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.263012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.267491] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.273417] ACPI: Interpreter enabled [ 0.275049] ACPI: PM: (supports S0 S3 S4 S5) [ 0.276019] ACPI: Using IOAPIC for interrupt routing [ 0.278113] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.281475] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.291135] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.292047] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.294024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.297089] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.302421] acpiphp: Slot [2] registered [ 0.304193] acpiphp: Slot [3] registered [ 0.305143] acpiphp: Slot [4] registered [ 0.306081] acpiphp: Slot [5] registered [ 0.307154] acpiphp: Slot [6] registered [ 0.308144] acpiphp: Slot [7] registered [ 0.310122] acpiphp: Slot [8] registered [ 0.311131] acpiphp: Slot [9] registered [ 0.312133] acpiphp: Slot [10] registered [ 0.314180] acpiphp: Slot [11] registered [ 0.315119] acpiphp: Slot [12] registered [ 0.316115] acpiphp: Slot [13] registered [ 0.318133] acpiphp: Slot [14] registered [ 0.320149] acpiphp: Slot [15] registered [ 0.321075] acpiphp: Slot [16] registered [ 0.323156] acpiphp: Slot [17] registered [ 0.324097] acpiphp: Slot [18] registered [ 0.325063] acpiphp: Slot [19] registered [ 0.326053] acpiphp: Slot [20] registered [ 0.326857] acpiphp: Slot [21] registered [ 0.328060] acpiphp: Slot [22] registered [ 0.329086] acpiphp: Slot [23] registered [ 0.331078] acpiphp: Slot [24] registered [ 0.332078] acpiphp: Slot [25] registered [ 0.333092] acpiphp: Slot [26] registered [ 0.334120] acpiphp: Slot [27] registered [ 0.336092] acpiphp: Slot [28] registered [ 0.337087] acpiphp: Slot [29] registered [ 0.339108] acpiphp: Slot [30] registered [ 0.340000] acpiphp: Slot [31] registered [ 0.340000] PCI host bridge to bus 0000:00 [ 0.340020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.342033] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.344018] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.346021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.348017] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.350017] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.351180] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.354466] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.359443] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.369021] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.373059] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.376014] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.377016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.379041] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.382172] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.384799] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.387052] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.390416] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.395023] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.409014] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.413016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.418909] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.425016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.429018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.447017] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.462485] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.468012] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.481025] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.498020] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.511627] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.518020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.527025] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.547020] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.557221] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.564028] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.572021] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.594018] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.604290] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.616037] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.623024] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.642016] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.662166] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.672024] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.680026] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.706033] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.716957] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.719386] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.721268] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.724349] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.726245] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.730125] iommu: Default domain type: Passthrough [ 0.732337] SCSI subsystem initialized [ 0.733129] ACPI: bus type USB registered [ 0.735119] usbcore: registered new interface driver usbfs [ 0.736071] usbcore: registered new interface driver hub [ 0.738139] usbcore: registered new device driver usb [ 0.740173] pps_core: LinuxPPS API ver. 1 registered [ 0.742015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.745047] PTP clock support registered [ 0.746143] EDAC MC: Ver: 3.0.0 [ 0.748154] PCI: Using ACPI for IRQ routing [ 0.750943] NetLabel: Initializing [ 0.752013] NetLabel: domain hash size = 128 [ 0.753010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.755106] NetLabel: unlabeled traffic allowed by default [ 0.757147] vgaarb: loaded [ 0.759314] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.760018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.765323] clocksource: Switched to clocksource kvm-clock [ 0.864790] VFS: Disk quotas dquot_6.6.0 [ 0.865637] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.867298] *** VALIDATE ramfs *** [ 0.868051] *** VALIDATE hugetlbfs *** [ 0.869151] pnp: PnP ACPI init [ 0.871154] pnp: PnP ACPI: found 6 devices [ 0.888642] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.892660] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.895472] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.898022] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.900827] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.903681] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.906772] NET: Registered protocol family 2 [ 0.909411] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.914203] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.917139] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.920739] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.923293] TCP: Hash tables configured (established 65536 bind 65536) [ 0.926253] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.929343] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.931536] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.935103] NET: Registered protocol family 1 [ 0.939035] RPC: Registered named UNIX socket transport module. [ 0.940789] RPC: Registered udp transport module. [ 0.942723] RPC: Registered tcp transport module. [ 0.944125] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.946575] NET: Registered protocol family 44 [ 0.948753] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.951199] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.953637] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.956639] PCI: CLS 0 bytes, default 64 [ 0.958286] Unpacking initramfs... [ 2.352179] debug: unmapping init [mem 0xffff8e0c3cc54000-0xffff8e0c3ffbffff] [ 2.355528] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.358023] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.360917] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.835815] Initialise system trusted keyrings [ 2.837496] Key type blacklist registered [ 2.840331] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.850899] zbud: loaded [ 2.853943] *** VALIDATE nfs *** [ 2.855913] *** VALIDATE nfs4 *** [ 2.857822] pstore: using deflate compression [ 2.861976] Platform Keyring initialized [ 2.970684] NET: Registered protocol family 38 [ 2.972523] Key type asymmetric registered [ 2.973852] Asymmetric key parser 'x509' registered [ 2.975258] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.978380] io scheduler mq-deadline registered [ 2.980253] io scheduler kyber registered [ 2.982033] io scheduler bfq registered [ 2.985172] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.987538] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.989600] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.992281] ACPI: Power Button [PWRF] [ 3.077927] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.162219] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.339935] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.436268] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.608059] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.636886] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.667923] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.674791] Non-volatile memory driver v1.3 [ 3.676541] Linux agpgart interface v0.103 [ 3.707908] virtio_blk virtio1: [vda] 132416 512-byte logical blocks (67.8 MB/64.7 MiB) [ 3.710727] vda: detected capacity change from 0 to 67796992 [ 3.726206] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.729303] vdb: detected capacity change from 0 to 1073741824 [ 3.742492] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.745586] vdc: detected capacity change from 0 to 2621440000 [ 3.763635] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.766488] vdd: detected capacity change from 0 to 2621440000 [ 3.780796] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.783267] vde: detected capacity change from 0 to 4294967296 [ 3.796157] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.798402] vdf: detected capacity change from 0 to 4294967296 [ 3.805813] libphy: Fixed MDIO Bus: probed [ 3.812847] usbcore: registered new interface driver usbserial_generic [ 3.815468] usbserial: USB Serial support registered for generic [ 3.817521] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.821638] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.823557] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.826107] mousedev: PS/2 mouse device common for all mice [ 3.829222] rtc_cmos 00:05: RTC can wake from S4 [ 3.832112] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.833463] rtc_cmos 00:05: registered as rtc0 [ 3.838600] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.839409] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.841570] intel_pstate: CPU model not supported [ 3.846733] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.852068] hid: raw HID events driver (C) Jiri Kosina [ 3.853883] usbcore: registered new interface driver usbhid [ 3.855503] usbhid: USB HID core driver [ 3.856855] drop_monitor: Initializing network drop monitor service [ 3.858848] Initializing XFRM netlink socket [ 3.860527] NET: Registered protocol family 10 [ 3.863429] Segment Routing with IPv6 [ 3.864778] NET: Registered protocol family 17 [ 3.866283] mpls_gso: MPLS GSO support [ 3.871485] RAS: Correctable Errors collector initialized. [ 3.873731] AVX version of gcm_enc/dec engaged. [ 3.875507] AES CTR mode by8 optimization enabled [ 3.947756] sched_clock: Marking stable (3947711998, 0)->(4836349139, -888637141) [ 3.951175] registered taskstats version 1 [ 3.953427] Loading compiled-in X.509 certificates [ 3.955625] zswap: loaded using pool lzo/zbud [ 3.981211] Key type big_key registered [ 3.994811] Key type encrypted registered [ 3.996338] ima: No TPM chip found, activating TPM-bypass! [ 3.998346] ima: Allocated hash algorithm: sha1 [ 3.999827] ima: No architecture policies found [ 4.001430] evm: Initialising EVM extended attributes: [ 4.003399] evm: security.selinux [ 4.004589] evm: security.ima [ 4.011086] evm: security.capability [ 4.013667] evm: HMAC attrs: 0x1 [ 4.021181] rtc_cmos 00:05: setting system clock to 2025-07-25 10:47:21 UTC (1753440441) [ 4.027852] debug: unmapping init [mem 0xffffffff89c03000-0xffffffff89dfffff] [ 4.030807] debug: unmapping init [mem 0xffffffff88982000-0xffffffff88c58fff] [ 4.040300] Write protecting the kernel read-only data: 28672k [ 4.043610] debug: unmapping init [mem 0xffffffff87003000-0xffffffff871fffff] [ 4.046560] debug: unmapping init [mem 0xffffffff87914000-0xffffffff879fffff] [ 4.079239] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.087732] systemd[1]: Detected virtualization kvm. [ 4.089436] systemd[1]: Detected architecture x86-64. [ 4.090704] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.117507] systemd[1]: No hostname configured. [ 4.119253] systemd[1]: Set hostname to . [ 4.121263] random: systemd: uninitialized urandom read (16 bytes read) [ 4.123389] systemd[1]: Initializing machine ID from random generator. [ 4.166592] random: ln: uninitialized urandom read (6 bytes read) [ 4.258597] random: systemd: uninitialized urandom read (16 bytes read) [ 4.261142] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.265374] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.268672] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.846696] device-mapper: uevent: version 1.0.3 [ 4.847982] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... [ 5.182147] random: fast init done Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.472345] virtio_net virtio0 ens2: renamed from eth0 [ 5.506214] scsi host0: ata_piix [ 5.546074] scsi host1: ata_piix [ 5.555198] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.557557] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.264462] dracut-initqueue[581]: RTNETLINK answers: File exists [ 10.527182] random: crng init done [ 10.528281] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.838680] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.923738] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.190871] SELinux: Disabled at runtime. [ 12.259520] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.267293] systemd[1]: Detected virtualization kvm. [ 12.268615] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.825230] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.828187] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.832442] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.835891] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.840309] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.851793] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.870935] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting udev Coldplug all Devices... Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Journal Service. [ 13.093558] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.362853] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.708719] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.747500] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.875896] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.909719] EDAC sbridge: Ver: 1.1.2 [ 15.808594] Key type dns_resolver registered [ 16.149962] NFS: Registering the id_resolver key type [ 16.151706] Key type id_resolver registered [ 16.153400] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg323-server login: [ 46.762628] libcfs: loading out-of-tree module taints kernel. [ 46.778535] Key type ._llcrypt registered [ 46.780843] Key type .llcrypt registered [ 46.845524] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 55.465409] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 56.070254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 56.078844] alg: No test for adler32 (adler32-zlib) [ 57.151941] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 57.545320] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 57.548153] LNet: Accept secure, port 988 [ 59.191232] Key type lgssc registered [ 59.917577] Lustre: Echo OBD driver; http://www.lustre.org/ [ 67.320677] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 68.426389] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 72.352880] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 75.123693] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 77.862069] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 83.910510] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 88.521147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 88.544715] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 88.557612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 89.664952] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 89.684303] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 89.734131] Lustre: lustre-MDT0000: new disk, initializing [ 89.795119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 89.806528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 91.331725] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 97.580844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 97.629567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 97.666176] Lustre: 6444:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 97.684649] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 97.687031] Lustre: Skipped 1 previous similar message [ 97.732828] Lustre: lustre-MDT0001: new disk, initializing [ 97.758715] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 97.771531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 97.776382] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 99.281726] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 101.782472] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 105.704497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 105.753799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 105.862952] Lustre: lustre-OST0000: new disk, initializing [ 105.866392] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 105.891907] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 108.245715] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 111.607123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 111.611885] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 111.650100] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 115.371716] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 115.406973] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 115.451496] Lustre: lustre-OST0001: new disk, initializing [ 115.453879] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 115.477703] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 116.999219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 117.002804] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 117.042983] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 117.922059] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 124.295761] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 131.301199] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 138.503196] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing check_logdir /tmp/testlogs/ [ 140.222752] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing yml_node [ 142.249964] Lustre: DEBUG MARKER: Client: 2.16.56.57 [ 143.265761] Lustre: DEBUG MARKER: MDS: 2.16.56.57 [ 144.344567] Lustre: DEBUG MARKER: OSS: 2.16.56.57 [ 145.052148] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Jul 25 06:49:41 EDT 2025 [ 152.031794] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 152.599821] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 168.415915] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 168.416588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 168.420485] Lustre: Skipped 3 previous similar messages [ 168.424559] Lustre: Skipped 3 previous similar messages [ 170.553905] LustreError: 12105:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 170.578767] Lustre: server umount lustre-MDT0000 complete [ 171.935900] LustreError: 6435:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753440609 with bad export cookie 3623653630617953508 [ 171.937153] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 171.940350] LustreError: 6435:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 173.535787] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 173.536053] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 173.541216] Lustre: Skipped 1 previous similar message [ 176.543714] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 176.546409] Lustre: Skipped 2 previous similar messages [ 181.663746] LustreError: 9585:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 181.663829] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 181.669268] LustreError: 9585:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 6 previous similar messages [ 181.672448] Lustre: Skipped 1 previous similar message [ 186.335156] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 186.368947] LustreError: 12307:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 186.371645] LustreError: 12307:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 186.440583] Lustre: server umount lustre-MDT0001 complete [ 187.904738] LustreError: 12510:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 187.906897] LustreError: 12510:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 187.927176] Lustre: server umount lustre-OST0000 complete [ 189.287598] Lustre: server umount lustre-OST0001 complete [ 191.624701] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 195.211789] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 198.940995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 201.305847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 203.503103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 205.739794] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 208.235079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 208.260778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 208.352982] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 208.367024] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 208.404614] Lustre: lustre-MDT0000: new disk, initializing [ 208.432720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 208.441469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 209.943499] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 214.622882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 214.663564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 214.708623] Lustre: 14872:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 214.843672] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 216.458813] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 219.916971] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 221.376786] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 224.035898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 224.062177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 224.169416] Lustre: lustre-OST0000: new disk, initializing [ 224.170954] Lustre: Skipped 1 previous similar message [ 224.173271] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 224.177148] Lustre: Skipped 2 previous similar messages [ 224.218300] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 224.221457] Lustre: Skipped 1 previous similar message [ 226.175653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 226.179043] Lustre: Skipped 1 previous similar message [ 226.181459] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 226.219500] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 226.312461] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 229.544720] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 230.834290] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 230.908944] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 232.246182] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 232.322879] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 236.514282] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 236.516927] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 236.520360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 236.523213] LustreError: Skipped 1 previous similar message [ 246.755279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 246.757516] Lustre: Skipped 4 previous similar messages [ 247.263117] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 247.282840] LustreError: 17778:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 247.284524] LustreError: 17778:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 247.343826] Lustre: server umount lustre-OST0000 complete [ 250.847994] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 250.850815] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 250.855402] Lustre: Skipped 1 previous similar message [ 254.726872] LustreError: 17981:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 254.728896] LustreError: 17981:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 254.764711] Lustre: server umount lustre-MDT0000 complete [ 256.081120] LustreError: 14864:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753440693 with bad export cookie 3623653630617955041 [ 256.084282] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 256.085027] LustreError: 14864:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 256.236515] Lustre: server umount lustre-MDT0001 complete [ 257.771507] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 06:51:34 (1753440694) [ 258.681406] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 259.319978] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 06:51:35 (1753440695) [ 261.725215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 261.914475] LustreError: 18850:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 261.942864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 263.312347] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 266.531833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 267.987761] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 269.954888] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 271.222772] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 273.630218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 275.771386] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 277.782838] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 292.322546] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 292.323125] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 292.326313] Lustre: Skipped 1 previous similar message [ 292.330307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 292.331958] Lustre: Skipped 4 previous similar messages [ 297.196185] LustreError: 21096:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 297.199307] LustreError: 21096:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 297.267349] Lustre: server umount lustre-OST0000 complete [ 302.562812] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 302.563421] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 302.566829] LustreError: Skipped 1 previous similar message [ 302.572160] Lustre: Skipped 2 previous similar messages [ 304.823433] Lustre: server umount lustre-MDT0000 complete [ 306.374243] LustreError: 18830:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753440743 with bad export cookie 3623653630617955923 [ 306.377112] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 306.378807] LustreError: 18830:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 306.533653] Lustre: server umount lustre-MDT0001 complete [ 312.079412] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 313.324901] Key type lgssc unregistered [ 313.490093] LNet: 22028:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 314.532837] LNet: Removed LNI 192.168.203.123@tcp [ 314.937683] Key type .llcrypt unregistered [ 314.938810] Key type ._llcrypt unregistered [ 325.317847] Key type ._llcrypt registered [ 325.319141] Key type .llcrypt registered [ 325.367518] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 325.824438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 325.832140] alg: No test for adler32 (adler32-zlib) [ 326.712793] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 326.812099] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 326.814234] LNet: Accept secure, port 988 [ 328.399100] Key type lgssc registered [ 328.822283] Lustre: Echo OBD driver; http://www.lustre.org/ [ 332.067194] loop: module loaded [ 332.087378] loop0: detected capacity change from 0 to 204800000 [ 332.153624] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 332.160139] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 332.292994] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 332.299260] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 332.326415] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 332.350444] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 332.366585] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 332.391594] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 334.608430] loop0: detected capacity change from 0 to 204800000 [ 334.685417] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 334.691290] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 334.699126] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 334.755438] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 336.699337] loop0: detected capacity change from 0 to 204800000 [ 339.032664] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 339.056615] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 339.066366] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 340.167203] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 340.184544] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 340.224928] Lustre: lustre-MDT0000: new disk, initializing [ 340.255817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 340.263351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 341.764820] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 345.770510] loop1: detected capacity change from 0 to 204800000 [ 347.907403] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 347.928388] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 348.039897] Lustre: lustre-OST07c6: new disk, initializing [ 348.042886] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 348.045523] Lustre: Skipped 1 previous similar message [ 348.077656] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 350.113783] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 350.785984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 350.789637] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 350.803277] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 437.728695] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 437.728813] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 437.736369] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 439.018127] LustreError: 26334:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 439.081809] Lustre: server umount lustre-OST07c6 complete [ 442.289886] LustreError: 26835:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 442.293104] LustreError: 26835:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 442.372309] Lustre: server umount lustre-MDT0000 complete [ 449.201389] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 450.299120] Key type lgssc unregistered [ 450.432458] LNet: 27661:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 451.493303] LNet: Removed LNI 192.168.203.123@tcp [ 451.820778] Key type .llcrypt unregistered [ 451.822228] Key type ._llcrypt unregistered [ 459.164705] Key type ._llcrypt registered [ 459.166084] Key type .llcrypt registered [ 459.201558] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 459.533277] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 459.597649] alg: No test for adler32 (adler32-zlib) [ 460.447894] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 460.539238] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 460.541393] LNet: Accept secure, port 988 [ 462.127154] Key type lgssc registered [ 462.511571] Lustre: Echo OBD driver; http://www.lustre.org/ [ 465.290760] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 465.301301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 466.487264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 467.925423] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 470.972961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 471.101317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 472.451699] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 474.517188] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 475.866982] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 478.379048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 478.508637] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 480.673198] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 482.675320] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 491.812887] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 495.017580] LustreError: 31079:0:(qsd_reint.c:617:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:2 until pending updates are flushed. [ 500.705860] LustreError: lustre-OST0000-osc-MDT0000: operation ost_setattr to node 0@lo failed: rc = -107 [ 500.705860] LustreError: lustre-OST0000-osc-MDT0000: operation ost_setattr to node 0@lo failed: rc = -107 [ 500.705886] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 500.717694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 501.216054] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 501.219293] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 501.225628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 502.241830] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 502.243961] Lustre: Skipped 1 previous similar message [ 503.981077] LustreError: 31330:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 504.044573] Lustre: server umount lustre-OST0000 complete [ 506.848098] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 506.850948] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 506.856315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 511.611310] LustreError: 31531:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 511.613289] LustreError: 31531:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 511.636816] Lustre: server umount lustre-MDT0000 complete [ 512.479970] LustreError: 28879:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 512.485233] LustreError: 28879:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 512.871252] LustreError: 28860:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753440950 with bad export cookie 472248910419819668 [ 512.874564] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 512.875588] LustreError: 28860:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 512.946559] LustreError: 31734:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 512.949280] LustreError: 31734:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 513.024235] Lustre: server umount lustre-MDT0001 complete [ 517.803919] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 518.949408] Key type lgssc unregistered [ 519.088443] LNet: 32261:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 520.100491] LNet: Removed LNI 192.168.203.123@tcp [ 520.437141] Key type .llcrypt unregistered [ 520.438676] Key type ._llcrypt unregistered [ 527.967867] Key type ._llcrypt registered [ 527.969126] Key type .llcrypt registered [ 528.012510] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 528.318825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 528.341376] alg: No test for adler32 (adler32-zlib) [ 529.208456] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 529.310506] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 529.312776] LNet: Accept secure, port 988 [ 530.903173] Key type lgssc registered [ 531.296489] Lustre: Echo OBD driver; http://www.lustre.org/ [ 534.099783] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 534.107259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 535.277884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 536.710095] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 539.857585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 540.010390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 541.472823] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 543.483274] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 544.769174] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 547.024642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 547.134760] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 548.951643] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 550.742442] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 560.486532] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 565.731095] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 565.732220] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 565.742085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 565.744889] Lustre: Skipped 1 previous similar message [ 567.022409] LustreError: 35744:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 567.089318] Lustre: server umount lustre-OST0000 complete [ 570.690756] LustreError: 35945:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 570.694294] LustreError: 35945:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 570.724655] Lustre: server umount lustre-MDT0000 complete [ 570.848809] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 570.849269] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 570.850750] LustreError: 33491:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 570.850758] LustreError: 33491:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 570.852777] LustreError: Skipped 1 previous similar message [ 570.857979] Lustre: Skipped 2 previous similar messages [ 572.365234] LustreError: 33475:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753441009 with bad export cookie 3859127201431791406 [ 572.370393] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 572.371777] LustreError: 33475:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 572.436942] LustreError: 36147:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 572.438821] LustreError: 36147:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 572.534654] Lustre: server umount lustre-MDT0001 complete [ 579.713872] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 582.893272] Key type lgssc unregistered [ 583.228943] LNet: 36673:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 584.297360] LNet: Removed LNI 192.168.203.123@tcp [ 585.203912] Key type .llcrypt unregistered [ 585.214950] Key type ._llcrypt unregistered [ 605.564173] hrtimer: interrupt took 5159585 ns [ 608.276643] Key type ._llcrypt registered [ 608.277990] Key type .llcrypt registered [ 608.380656] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 609.562567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 609.655074] alg: No test for adler32 (adler32-zlib) [ 610.965920] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 611.292489] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 611.296933] LNet: Accept secure, port 988 [ 613.007151] Key type lgssc registered [ 614.362408] Lustre: Echo OBD driver; http://www.lustre.org/ [ 623.416416] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 623.468178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 625.464172] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 631.449695] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 634.851612] LustreError: 38060:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 634.886079] LustreError: 38060:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 639.977370] LustreError: 38060:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 642.330517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 642.833442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 646.745626] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 651.495988] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 654.610251] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 660.615241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 660.888722] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 665.756865] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 670.143839] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 670.220409] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 677.856656] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.23@tcp (stopping) [ 678.121174] LustreError: 40291:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 678.216718] Lustre: server umount lustre-MDT0000 complete [ 678.884350] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 678.900100] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 678.923485] LustreError: 38061:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 680.418513] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 681.222622] LustreError: 38040:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753441118 with bad export cookie 4980933650825783296 [ 681.231147] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 682.960177] LustreError: 39092:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 682.977132] LustreError: 39092:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 682.994921] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.23@tcp (stopping) [ 685.537923] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 685.546441] Lustre: Skipped 1 previous similar message [ 685.555879] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 687.490480] LustreError: 40492:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 687.502932] LustreError: 40492:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 687.710253] Lustre: server umount lustre-MDT0001 complete [ 716.265071] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 716.313227] LustreError: 40715:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 716.315426] LustreError: 40715:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 716.452321] Lustre: server umount lustre-OST0000 complete [ 730.481180] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 733.308300] Key type lgssc unregistered [ 733.588690] LNet: 41548:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 734.631698] LNet: Removed LNI 192.168.203.123@tcp [ 735.483797] Key type .llcrypt unregistered [ 735.486987] Key type ._llcrypt unregistered [ 753.337410] Key type ._llcrypt registered [ 753.338828] Key type .llcrypt registered [ 753.419128] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 754.128728] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 754.178628] alg: No test for adler32 (adler32-zlib) [ 755.177496] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 755.328796] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 755.331394] LNet: Accept secure, port 988 [ 756.959653] Key type lgssc registered [ 757.797147] Lustre: Echo OBD driver; http://www.lustre.org/ [ 763.975321] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 763.990802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 765.350224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 768.121465] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 773.816468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 774.083950] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 776.802392] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 780.395316] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 782.306953] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 786.050832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 786.254550] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 790.380031] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 790.507737] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 795.369295] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 805.856198] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 805.866268] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 805.886440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 808.425303] Lustre: lustre-OST0000: Not available for connect from 192.168.203.23@tcp (stopping) [ 808.933272] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 808.937544] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 813.518996] Lustre: lustre-OST0000: Not available for connect from 192.168.203.23@tcp (stopping) [ 813.523695] Lustre: Skipped 2 previous similar messages [ 818.640180] Lustre: lustre-OST0000: Not available for connect from 192.168.203.23@tcp (stopping) [ 818.645177] Lustre: Skipped 2 previous similar messages [ 819.170054] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 819.207616] LustreError: 45139:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 819.299862] Lustre: server umount lustre-OST0000 complete [ 834.528826] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 834.537296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 834.540054] Lustre: Skipped 2 previous similar messages [ 840.600977] LustreError: 45529:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 840.606009] LustreError: 45529:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 840.672410] Lustre: server umount lustre-MDT0000 complete [ 843.504465] LustreError: 42894:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753441280 with bad export cookie 4850691437788779035 [ 843.509436] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 843.509936] LustreError: 42894:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message [ 849.673094] LustreError: 45733:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 849.675862] LustreError: 45733:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 849.850805] Lustre: server umount lustre-MDT0001 complete [ 858.713544] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 861.120906] Key type lgssc unregistered [ 861.350525] LNet: 46260:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 862.378340] LNet: Removed LNI 192.168.203.123@tcp [ 863.059302] Key type .llcrypt unregistered [ 863.062606] Key type ._llcrypt unregistered [ 881.243723] Key type ._llcrypt registered [ 881.245607] Key type .llcrypt registered [ 881.359943] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 882.107071] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 882.135184] alg: No test for adler32 (adler32-zlib) [ 883.148314] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 883.315787] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 883.318293] LNet: Accept secure, port 988 [ 884.938710] Key type lgssc registered [ 885.805791] Lustre: Echo OBD driver; http://www.lustre.org/ [ 892.845379] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 892.859414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 894.196358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 897.712467] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 903.651041] LustreError: 47624:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 903.659470] LustreError: 47624:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 904.046823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 904.333584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 907.456859] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 911.782259] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 914.845906] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 921.025196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 921.477573] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 926.211458] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 930.682317] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 938.936322] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 946.692496] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 949.808446] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 957.429784] LNetError: Refusing connection from 192.168.203.123 for 127.0.0.2@tcp: No matching NI [ 957.451724] LNetError: 46926:0:(socklnd_cb.c:1794:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 957.468959] LNetError: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 961.437687] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2025-07-25 7h03m17s [ 962.714383] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 964.580088] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 964.581607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 964.595122] Lustre: Skipped 2 previous similar messages [ 964.598466] Lustre: Skipped 1 previous similar message [ 964.621230] LNetError: Refusing connection from 192.168.203.123 for 127.0.0.2@tcp: No matching NI [ 964.627092] LNetError: 46929:0:(socklnd_cb.c:1794:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 964.630676] LNetError: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 965.670842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 965.675327] Lustre: Skipped 3 previous similar messages [ 968.185099] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.23@tcp (stopping) [ 970.345377] LustreError: 50192:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 970.400432] Lustre: server umount lustre-MDT0000 complete [ 970.722060] LNetError: Refusing connection from 192.168.203.123 for 127.0.0.2@tcp: No matching NI [ 970.741116] LNetError: 46927:0:(socklnd_cb.c:1794:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 970.752160] LNetError: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 971.754323] LustreError: 47624:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 971.765273] LustreError: 47624:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 973.037172] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 974.287399] LustreError: 47618:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 976.868263] LNetError: Refusing connection from 192.168.203.123 for 127.0.0.2@tcp: No matching NI [ 976.873564] LNetError: 46928:0:(socklnd_cb.c:1794:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 976.881575] LNetError: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 977.896796] LustreError: 47624:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 977.909894] LustreError: 47624:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 978.023963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 978.161625] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 978.346213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 978.394889] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 979.378847] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 980.982338] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 983.586515] Lustre: DEBUG MARKER: Wait for df (36050) ... [ 984.833444] Lustre: DEBUG MARKER: done [ 988.641524] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 988.645269] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 988.653691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 988.657094] Lustre: Skipped 1 previous similar message [ 989.668272] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 993.264099] LustreError: 51241:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 993.266321] LustreError: 51241:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 993.417591] Lustre: server umount lustre-OST0000 complete [ 999.905796] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 999.906307] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 999.907746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 999.907752] Lustre: Skipped 2 previous similar messages [ 999.933655] Lustre: Skipped 1 previous similar message [ 999.943875] LNetError: Refusing connection from 192.168.203.123 for 127.0.0.2@tcp: No matching NI [ 999.949862] LNetError: 46926:0:(socklnd_cb.c:1794:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 999.956609] LNetError: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1002.585605] LustreError: 51444:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1002.587741] LustreError: 51444:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1002.637423] Lustre: server umount lustre-MDT0000 complete [ 1005.251913] LustreError: 47604:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753441442 with bad export cookie 3467769327707447158 [ 1005.255058] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1005.261242] LustreError: 47604:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1005.388852] LustreError: 51645:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1005.395621] LustreError: 51645:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1005.563330] Lustre: server umount lustre-MDT0001 complete [ 1014.460859] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 1016.451368] Key type lgssc unregistered [ 1016.695682] LNet: 52172:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1017.765772] LNet: Removed LNI 192.168.203.123@tcp [ 1018.410914] Key type .llcrypt unregistered [ 1018.412694] Key type ._llcrypt unregistered [ 1044.339311] Key type ._llcrypt registered [ 1044.340973] Key type .llcrypt registered [ 1044.429253] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 1054.607797] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 1055.284197] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1055.405140] alg: No test for adler32 (adler32-zlib) [ 1056.434947] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 1056.615233] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 1056.620597] LNet: Accept secure, port 988 [ 1058.279267] Key type lgssc registered [ 1059.199485] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1067.069793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1072.508984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1077.735871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1083.053421] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1088.141227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1088.179134] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1088.189429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1089.414899] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1089.473653] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1089.542456] Lustre: lustre-MDT0000: new disk, initializing [ 1089.615391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1089.630188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1092.847982] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1101.952483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1102.024386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1102.085959] Lustre: 56084:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1102.136130] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1102.140582] Lustre: Skipped 1 previous similar message [ 1102.195500] Lustre: lustre-MDT0001: new disk, initializing [ 1102.272442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1102.295620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1102.312929] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1105.616966] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1111.112741] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1114.418152] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1120.542924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1120.610172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1120.821361] Lustre: lustre-OST0000: new disk, initializing [ 1120.827218] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1120.925160] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1125.650223] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1130.543022] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1130.563951] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1130.667268] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1131.854281] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1134.836391] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1134.967072] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1137.504052] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1137.704310] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1141.220509] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1141.228924] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1141.235491] LustreError: Skipped 1 previous similar message [ 1141.243556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1141.249917] Lustre: Skipped 1 previous similar message [ 1145.212152] LustreError: 58990:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1145.361570] Lustre: server umount lustre-OST0000 complete [ 1148.387380] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1148.390804] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1148.397975] Lustre: Skipped 1 previous similar message [ 1148.402592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1151.459695] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1151.460477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1153.880643] LustreError: 59192:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1153.885897] LustreError: 59192:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1153.983388] Lustre: server umount lustre-MDT0000 complete [ 1156.205729] LustreError: 56077:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753441593 with bad export cookie 5283300435144596562 [ 1156.211151] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1156.307571] LustreError: 59394:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1156.312045] LustreError: 59394:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1156.459871] Lustre: server umount lustre-MDT0001 complete [ 1161.464843] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 07:06:37 (1753441597) [ 1165.942147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1166.343946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1168.853481] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1171.425099] LustreError: 60064:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1171.442151] LustreError: 60064:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1174.897244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1175.170518] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1177.955234] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1181.860766] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1184.233531] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1188.584205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1188.799124] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1192.151245] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1195.638457] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1204.987137] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2025-07-25 7h07m21s [ 1206.063163] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1209.951698] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1211.336855] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1211.375251] LustreError: 60059:0:(mdt_handler.c:2333:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1215.980665] LNetError: Refusing connection from 192.168.203.123 for 127.0.0.2@tcp: No matching NI [ 1215.984712] LNetError: 54287:0:(socklnd_cb.c:1794:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1215.987438] LNetError: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1231.851074] Lustre: lustre-MDT0000: Client 055e483d-072d-4de4-a824-af3613d75425 (at 192.168.203.23@tcp) reconnecting [ 1251.455135] LustreError: 60059:0:(mdt_handler.c:2333:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1253.048760] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1254.670903] Lustre: DEBUG MARKER: done [ 1261.540570] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1261.541636] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1261.550618] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1261.554244] Lustre: Skipped 2 previous similar messages [ 1263.859987] LustreError: 63044:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1263.864280] LustreError: 63044:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1263.984574] Lustre: server umount lustre-OST0000 complete [ 1266.659534] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1266.662428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1266.665058] Lustre: Skipped 2 previous similar messages [ 1266.673512] LNetError: Refusing connection from 192.168.203.123 for 127.0.0.2@tcp: No matching NI [ 1266.681755] Lustre: Skipped 1 previous similar message [ 1266.690162] LNetError: Skipped 1 previous similar message [ 1266.690560] LNetError: 54288:0:(socklnd_cb.c:1794:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1266.690567] LNetError: 54288:0:(socklnd_cb.c:1794:ksocknal_recv_hello()) Skipped 1 previous similar message [ 1266.690620] LNetError: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1266.690623] LNetError: Skipped 1 previous similar message [ 1271.777332] LNetError: Refusing connection from 192.168.203.123 for 127.0.0.2@tcp: No matching NI [ 1271.783414] LNetError: 54289:0:(socklnd_cb.c:1794:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1271.787746] LNetError: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1272.394528] LustreError: 63249:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1272.397014] LustreError: 63249:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1272.451225] Lustre: server umount lustre-MDT0000 complete [ 1272.801086] LustreError: 60064:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1272.809327] LustreError: 60064:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 1274.460155] LustreError: 60045:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753441711 with bad export cookie 5283300435144597437 [ 1274.463818] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1274.487146] LustreError: 60045:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1274.685951] Lustre: server umount lustre-MDT0001 complete [ 1281.517557] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 1283.491847] Key type lgssc unregistered [ 1283.740563] LNet: 63977:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1284.778258] LNet: Removed LNI 192.168.203.123@tcp [ 1285.352482] Key type .llcrypt unregistered [ 1285.354358] Key type ._llcrypt unregistered [ 1306.573887] Key type ._llcrypt registered [ 1306.575400] Key type .llcrypt registered [ 1306.644530] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 1316.560907] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 1317.286195] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1317.313084] alg: No test for adler32 (adler32-zlib) [ 1318.280973] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 1318.437230] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 1318.439970] LNet: Accept secure, port 988 [ 1320.095198] Key type lgssc registered [ 1320.863184] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1327.404015] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1332.231419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1336.455028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1340.892339] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1345.333836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1345.383852] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1345.395863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1346.520194] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1346.548409] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1346.603700] Lustre: lustre-MDT0000: new disk, initializing [ 1346.673487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1346.687409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1349.024533] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1355.161849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1355.200238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1355.252717] Lustre: 67890:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1355.293403] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1355.296467] Lustre: Skipped 1 previous similar message [ 1355.342815] Lustre: lustre-MDT0001: new disk, initializing [ 1355.411722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1355.443861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1355.448923] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1357.451549] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1361.832631] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1363.563210] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1366.879752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1366.923380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1367.069970] Lustre: lustre-OST0000: new disk, initializing [ 1367.074297] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1367.127519] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1369.561909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1369.572122] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1369.639966] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1370.246816] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1374.475249] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1376.382366] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1376.511488] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1378.409874] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1378.514176] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1379.808303] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1379.811892] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1379.813455] LustreError: Skipped 1 previous similar message [ 1379.821627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1379.824467] Lustre: Skipped 1 previous similar message [ 1384.930733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1384.934564] Lustre: Skipped 1 previous similar message [ 1385.518254] LustreError: 70796:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1385.612167] Lustre: server umount lustre-OST0000 complete [ 1390.052105] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1390.060987] Lustre: Skipped 1 previous similar message [ 1390.067448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1390.071139] Lustre: Skipped 1 previous similar message [ 1393.673747] LustreError: 70997:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1393.676576] LustreError: 70997:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1393.692201] LustreError: 70997:0:(ldlm_resource.c:981:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (0000000096b6defa) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 1393.731693] Lustre: server umount lustre-MDT0000 complete [ 1395.651646] LustreError: 68665:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753441833 with bad export cookie 6246127306507635822 [ 1395.655817] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1395.657595] LustreError: 68665:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1395.728779] LustreError: 71200:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1395.731306] LustreError: 71200:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1395.860642] Lustre: server umount lustre-MDT0001 complete [ 1399.994372] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 07:10:36 (1753441836) [ 1400.840992] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1401.908127] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 07:10:38 (1753441838) [ 1409.047494] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 1411.376719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1411.626113] LustreError: 72237:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1411.634820] LustreError: 72237:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 1411.668636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1412.214093] Lustre: Failing over lustre-MDT0000 [ 1412.218449] LustreError: 72265:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1412.223041] LustreError: 72265:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1412.226687] LustreError: 72265:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 0, retries 0, failed: rc = -5 [ 1412.315264] LustreError: 72316:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1412.319391] LustreError: 72316:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1412.466912] Lustre: server umount lustre-MDT0000 complete [ 1416.286619] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 07:10:52 (1753441852) [ 1420.449660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1420.697784] LustreError: 72885:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1420.736112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1422.695964] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1425.890764] LustreError: 72885:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1427.037377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1429.294937] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1432.274599] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1434.172972] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1437.710336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1437.922211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1437.925068] Lustre: Skipped 1 previous similar message [ 1441.255363] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1444.116654] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1456.782836] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1462.757268] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1462.759924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1462.769058] Lustre: Skipped 2 previous similar messages [ 1462.778672] Lustre: Skipped 2 previous similar messages [ 1466.148104] LustreError: 75230:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1466.152457] LustreError: 75230:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1466.220984] Lustre: server umount lustre-MDT0000 complete [ 1467.872580] LustreError: 73590:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1467.883236] LustreError: 73590:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1468.291426] LustreError: 72866:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753441905 with bad export cookie 6246127306507636977 [ 1468.296097] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1468.296277] LustreError: 72866:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1468.501616] Lustre: server umount lustre-MDT0001 complete [ 1470.356746] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1472.859191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1477.150171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1478.111761] LustreError: 66104:0:(client.c:1385:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ab1092e2 x1838616941755264/t0(0) o250->MGC192.168.203.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1478.304518] LustreError: 74480:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1478.385511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1478.466794] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1480.723758] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1483.750679] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1483.758829] Lustre: Skipped 1 previous similar message [ 1485.396813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1487.596956] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1489.696092] Lustre: 66107:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1753441911/real 1753441911] req@000000004d65f0b5 x1838616941755008/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1753441927 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1490.663039] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1490.921421] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 1492.961238] Lustre: 66105:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1753441914/real 1753441914] req@000000008d86fdd3 x1838616941755136/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1753441930 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1492.966165] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 1492.984121] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1493.000713] Lustre: 66105:0:(client.c:2453:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1494.047235] Lustre: 66105:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1753441915/real 1753441915] req@0000000039a489fd x1838616941755392/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1753441931 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1498.081374] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1498.085335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1498.092607] Lustre: Skipped 2 previous similar messages [ 1498.094318] Lustre: Skipped 1 previous similar message [ 1502.676593] LustreError: 77677:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1502.678618] LustreError: 77677:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1502.778173] Lustre: server umount lustre-MDT0000 complete [ 1503.200565] LustreError: 76288:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1503.206592] LustreError: 76288:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 1505.166663] LustreError: 72864:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753441942 with bad export cookie 6246127306507638881 [ 1505.169508] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1505.173200] LustreError: 72864:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1505.411840] Lustre: server umount lustre-MDT0001 complete [ 1509.226055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1514.692958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1515.488313] LustreError: 66104:0:(client.c:1385:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000614bc8af x1838616941776256/t0(0) o250->MGC192.168.203.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1515.706787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1515.709400] Lustre: Skipped 1 previous similar message [ 1515.748200] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1518.144346] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1521.129369] LustreError: 74479:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1521.133607] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1521.139828] LustreError: 74479:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 1523.462578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1524.640170] Lustre: 66108:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1753441945/real 1753441945] req@0000000001cf0f4a x1838616941775872/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1753441961 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1525.877558] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1527.783753] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 1528.573404] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1528.799408] Lustre: 66106:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1753441950/real 1753441950] req@00000000bb7306d9 x1838616941776128/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1753441966 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1528.814172] Lustre: 66106:0:(client.c:2453:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1530.328390] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1533.926236] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1533.931972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1538.019263] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1538.028079] Lustre: Skipped 2 previous similar messages [ 1539.273915] LustreError: 80117:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1539.276678] LustreError: 80117:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1539.331101] Lustre: server umount lustre-MDT0000 complete [ 1541.457075] LustreError: 72864:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753441978 with bad export cookie 6246127306507640253 [ 1541.461153] LustreError: 72864:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1541.461945] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1541.631641] Lustre: server umount lustre-MDT0001 complete [ 1545.109355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1547.012382] Lustre: DEBUG MARKER: files compared the same [ 1558.426076] Lustre: server umount lustre-OST0000 complete [ 1568.283848] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 1570.043897] Key type lgssc unregistered [ 1570.199952] LNet: 81744:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1571.237196] LNet: Removed LNI 192.168.203.123@tcp [ 1571.629222] Key type .llcrypt unregistered [ 1571.630390] Key type ._llcrypt unregistered [ 1584.293369] Key type ._llcrypt registered [ 1584.294658] Key type .llcrypt registered [ 1584.359280] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 1585.077986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1585.094181] alg: No test for adler32 (adler32-zlib) [ 1586.071346] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 1586.216582] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 1586.219716] LNet: Accept secure, port 988 [ 1587.839180] Key type lgssc registered [ 1588.575799] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1593.158027] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1593.171782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1594.451289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1596.718892] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug +malloc all [ 1602.073708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1602.258796] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1604.815738] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug +malloc all [ 1608.425128] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1610.752979] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1615.315140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1615.551415] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1618.610857] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1619.180402] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug +malloc all [ 1622.602338] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1632.226325] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1632.226732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1632.226847] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1632.234563] Lustre: Skipped 1 previous similar message [ 1633.211145] LustreError: 85344:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1633.310569] Lustre: server umount lustre-OST0000 complete [ 1637.344736] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1637.345047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1637.349592] Lustre: Skipped 1 previous similar message [ 1637.356692] Lustre: Skipped 1 previous similar message [ 1641.483017] LustreError: 85545:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1641.486215] LustreError: 85545:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1641.552301] Lustre: server umount lustre-MDT0000 complete [ 1642.466264] LustreError: 83110:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1642.477388] LustreError: 83110:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 1643.559864] LustreError: 85547:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753442081 with bad export cookie 5098028208293422662 [ 1643.562658] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1643.564741] LustreError: 85547:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1643.645899] LustreError: 85748:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1643.647932] LustreError: 85748:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1643.759911] Lustre: server umount lustre-MDT0001 complete [ 1650.293606] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 1652.082672] Key type lgssc unregistered [ 1652.326960] LNet: 86276:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1653.351229] LNet: Removed LNI 192.168.203.123@tcp [ 1653.870970] Key type .llcrypt unregistered [ 1653.873099] Key type ._llcrypt unregistered [ 1667.359721] Key type ._llcrypt registered [ 1667.361235] Key type .llcrypt registered [ 1667.419463] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 1667.917561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1667.981941] alg: No test for adler32 (adler32-zlib) [ 1668.895435] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 1669.017083] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 1669.021074] LNet: Accept secure, port 988 [ 1670.647177] Key type lgssc registered [ 1671.338032] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1676.198246] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1676.207417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1693.728083] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1699.871679] LustreError: 87613:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.123@tcp: failed processing log, type 4: rc = -110 [ 1730.591193] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1733.031459] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1735.481288] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1738.945907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1739.023775] Lustre: *** cfs_fail_loc=706, val=0*** [ 1739.235503] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1739.288707] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1741.030099] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1744.352525] LustreError: 87640:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1744.362217] LustreError: 87640:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 1744.755328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1744.945679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1746.613774] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1749.040133] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1750.872991] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1754.594201] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1754.597678] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1754.607837] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1755.108997] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1755.109797] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1755.120596] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1755.122694] Lustre: Skipped 1 previous similar message [ 1760.226823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1765.349159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1765.356245] Lustre: Skipped 1 previous similar message [ 1766.367122] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1766.393211] LustreError: 89915:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 1766.488782] Lustre: server umount lustre-OST0000 complete [ 1770.467499] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1770.470309] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1770.476619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1770.476627] Lustre: Skipped 1 previous similar message [ 1770.481138] Lustre: Skipped 1 previous similar message [ 1774.271303] LustreError: 90119:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1774.274240] LustreError: 90119:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1774.312933] Lustre: server umount lustre-MDT0000 complete [ 1775.584585] LustreError: 88582:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1775.593561] LustreError: 88582:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 1776.068528] LustreError: 87622:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753442213 with bad export cookie 8995458299059806879 [ 1776.072331] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1776.079533] LustreError: 87622:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1776.146708] LustreError: 90323:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1776.149909] LustreError: 90323:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1776.256828] Lustre: server umount lustre-MDT0001 complete [ 1780.761919] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 1782.131976] Key type lgssc unregistered [ 1782.284810] LNet: 90850:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1783.333467] LNet: Removed LNI 192.168.203.123@tcp [ 1783.763046] Key type .llcrypt unregistered [ 1783.765111] Key type ._llcrypt unregistered [ 1796.404962] Key type ._llcrypt registered [ 1796.406566] Key type .llcrypt registered [ 1796.471295] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 1797.075875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1797.098100] alg: No test for adler32 (adler32-zlib) [ 1798.019153] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 1798.172695] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 1798.175402] LNet: Accept secure, port 988 [ 1799.783164] Key type lgssc registered [ 1800.352176] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1804.258861] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1804.268741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1807.279861] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1811.435341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1811.756729] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1813.864869] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1817.058743] LustreError: 92877:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1817.066102] LustreError: 92877:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 1818.849381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1819.051857] LustreError: 92877:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1819.059501] LustreError: 92877:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 1819.069047] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1822.114599] LustreError: 92877:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1822.126670] LustreError: 92877:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1822.298480] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1827.008286] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1827.015684] Lustre: MGS: Client 4f523417-2c43-420f-96ab-7a5617a81dad (at 0@lo) reconnecting [ 1827.161563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1827.199798] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1829.332347] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1839.585995] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1839.595932] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1839.604380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1842.658819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1842.658845] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1842.668924] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1845.297152] LustreError: 94819:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1845.383667] Lustre: server umount lustre-OST0000 complete [ 1847.778556] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1847.784954] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1847.789400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1847.795669] Lustre: Skipped 2 previous similar messages [ 1852.897510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1852.901337] Lustre: Skipped 1 previous similar message [ 1853.321664] LustreError: 95021:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1853.324468] LustreError: 95021:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1853.343344] Lustre: server umount lustre-MDT0000 complete [ 1855.420301] LustreError: 95223:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1855.422445] LustreError: 95223:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1855.542481] Lustre: server umount lustre-MDT0001 complete [ 1857.518316] LustreError: 95423:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1857.520753] LustreError: 95423:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1857.560779] Lustre: server umount MGS complete [ 1863.703338] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 07:18:20 (1753442300) [ 1877.650335] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 1881.943776] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 1886.850068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1890.221452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1893.611753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1897.341773] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1901.562352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1901.614150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1903.877649] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1909.293594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1909.326780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1909.425598] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1909.452379] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1909.494280] Lustre: lustre-MDT0001: new disk, initializing [ 1909.533736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1909.547234] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1909.551223] Lustre: 99914:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1909.555825] LustreError: 99914:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1911.383846] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1917.272181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1917.320563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1917.454570] Lustre: lustre-OST0000: new disk, initializing [ 1917.458672] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1917.508546] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1920.349895] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1924.485359] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1924.641726] Lustre: 101206:0:(client.c:1611:after_reply()) @@@ resending request on EINPROGRESS req@00000000d0f4c4d1 x1838617445043200/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1753442378 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1927.686667] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1927.693406] Lustre: MGS: Client eeb8b683-bf91-4152-bcbf-7564f9c32392 (at 0@lo) reconnecting [ 1927.719793] Lustre: 99203:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1927.729323] LustreError: 99203:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 1927.745021] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1927.785529] Lustre: lustre-MDT0000: new disk, initializing [ 1927.848878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1927.861984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1928.928873] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1928.929046] Lustre: 101614:0:(client.c:1611:after_reply()) @@@ resending request on EINPROGRESS req@000000004477c06e x1838617445048576/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1753442382 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1928.940956] Lustre: 101614:0:(client.c:1611:after_reply()) Skipped 1 previous similar message [ 1929.808987] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1933.738985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1933.745582] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1933.993243] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 1958.371867] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1958.377509] Lustre: Skipped 2 previous similar messages [ 1958.378470] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1958.384158] Lustre: Skipped 1 previous similar message [ 1961.726309] LustreError: 102130:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 1961.729362] LustreError: 102130:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1961.839193] Lustre: server umount lustre-OST0000 complete [ 1968.610270] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1968.610312] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1968.613169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1968.627679] Lustre: Skipped 1 previous similar message [ 1970.253274] LustreError: 102331:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 1970.256108] LustreError: 102331:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1970.270580] Lustre: server umount lustre-MDT0000 complete [ 1978.780446] Lustre: server umount lustre-MDT0001 complete [ 1980.909906] Lustre: server umount MGS complete [ 1986.913908] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 07:20:23 (1753442423) [ 1996.668404] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 1998.302718] Key type lgssc unregistered [ 1998.470190] LNet: 104064:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1999.526479] LNet: Removed LNI 192.168.203.123@tcp [ 2000.058893] Key type .llcrypt unregistered [ 2000.060661] Key type ._llcrypt unregistered [ 2008.976139] Key type ._llcrypt registered [ 2008.977872] Key type .llcrypt registered [ 2009.032951] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2009.456323] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2009.508206] alg: No test for adler32 (adler32-zlib) [ 2010.418459] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 2010.545325] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 2010.550948] LNet: Accept secure, port 988 [ 2012.167177] Key type lgssc registered [ 2012.740537] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2016.724734] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2016.739554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2018.001193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2020.009973] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2024.754224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2024.965432] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2027.043493] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2030.082631] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2032.074596] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2035.937258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2036.171412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2038.900920] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2041.718752] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2043.367441] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 2055.649445] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2055.650413] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2055.653242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2055.661813] Lustre: Skipped 1 previous similar message [ 2057.269183] LustreError: 107479:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2057.350587] Lustre: server umount lustre-OST0000 complete [ 2060.768993] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2060.769536] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2060.771595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2060.771601] Lustre: Skipped 1 previous similar message [ 2060.783921] Lustre: Skipped 1 previous similar message [ 2065.282785] LustreError: 107680:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2065.285420] LustreError: 107680:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2065.319153] Lustre: server umount lustre-MDT0000 complete [ 2065.887957] LustreError: 107682:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2065.894284] LustreError: 107682:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 2067.030947] LustreError: 105945:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753442504 with bad export cookie 17474037989315818880 [ 2067.033116] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2067.037909] LustreError: 105945:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2067.111561] LustreError: 107883:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2067.113694] LustreError: 107883:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2067.215588] Lustre: server umount lustre-MDT0001 complete [ 2072.866878] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 2074.133700] Key type lgssc unregistered [ 2074.289558] LNet: 108410:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2075.302432] LNet: Removed LNI 192.168.203.123@tcp [ 2075.687220] Key type .llcrypt unregistered [ 2075.688687] Key type ._llcrypt unregistered [ 2083.525259] Key type ._llcrypt registered [ 2083.526437] Key type .llcrypt registered [ 2083.579499] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2084.021818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2084.046552] alg: No test for adler32 (adler32-zlib) [ 2084.964511] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 2085.085296] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 2085.088597] LNet: Accept secure, port 988 [ 2086.703161] Key type lgssc registered [ 2087.195830] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2090.011433] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2090.011649] LustreError: 109440:0:(tgt_mount.c:2228:server_fill_super()) cfs_race id 716 sleeping [ 2090.014817] Lustre: Skipped 1 previous similar message [ 2090.014836] LustreError: 109439:0:(tgt_mount.c:2228:server_fill_super()) cfs_fail_race id 716 waking [ 2090.025637] LustreError: 109440:0:(tgt_mount.c:2228:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 [ 2090.029896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2090.030809] LustreError: 109440:0:(tgt_mount.c:2240:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2090.039950] LustreError: 109440:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2091.311358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2094.903992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2095.067162] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2097.024973] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2099.339646] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing lsmod [ 2100.675689] LustreError: 110603:0:(tgt_mount.c:2228:server_fill_super()) cfs_race id 716 sleeping [ 2100.675795] LustreError: 110604:0:(tgt_mount.c:2228:server_fill_super()) cfs_fail_race id 716 waking [ 2100.683432] LustreError: 110603:0:(tgt_mount.c:2228:server_fill_super()) cfs_fail_race id 716 awake: rc=4996 [ 2100.687329] LustreError: 110603:0:(tgt_mount.c:2240:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2100.693838] LustreError: 110603:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2100.698448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2100.866500] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2101.926626] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 2105.314812] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2105.322210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2105.327716] Lustre: Skipped 2 previous similar messages [ 2105.332304] Lustre: Skipped 1 previous similar message [ 2105.338289] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2108.357058] LustreError: 110791:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2108.404680] Lustre: server umount lustre-MDT0000 complete [ 2110.104258] LustreError: 109458:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753442547 with bad export cookie 2928227316732081032 [ 2110.108190] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2110.109108] LustreError: 109458:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2110.437232] LustreError: 109475:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2110.442354] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2110.444062] LustreError: 109475:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 2110.454364] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2110.458457] Lustre: Skipped 1 previous similar message [ 2111.519250] LustreError: 109912:0:(client.c:1375:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000750c7a23 x1838617745966720/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2111.533437] LustreError: 110995:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2111.535811] LustreError: 110995:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2111.629974] Lustre: server umount lustre-MDT0001 complete [ 2113.390832] LustreError: 111197:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2113.393591] LustreError: 111197:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2113.495760] Lustre: server umount lustre-OST0000 complete [ 2117.452081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2117.666093] LustreError: 111672:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2117.699171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2119.379657] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2122.720457] LustreError: 111672:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2123.326802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2123.512851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2125.294115] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2127.904525] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2129.689734] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2133.123309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2133.325131] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2136.001331] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2138.251558] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2142.570367] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:97) [ 2152.932612] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2152.933656] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2152.936864] LustreError: Skipped 1 previous similar message [ 2152.938234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2152.953634] Lustre: Skipped 1 previous similar message [ 2157.550648] LustreError: 113917:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2157.553567] LustreError: 113917:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2157.632706] Lustre: server umount lustre-OST0000 complete [ 2159.583966] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2159.587105] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2159.592382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2159.595288] Lustre: Skipped 1 previous similar message [ 2165.634927] LustreError: 114120:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2165.637323] LustreError: 114120:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2165.687502] Lustre: server umount lustre-MDT0000 complete [ 2167.396337] LustreError: 111652:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753442604 with bad export cookie 2928227316732081949 [ 2167.400776] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2167.401653] LustreError: 111652:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2167.583688] Lustre: server umount lustre-MDT0001 complete [ 2173.159704] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 2174.627196] Key type lgssc unregistered [ 2174.786859] LNet: 114848:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2175.844555] LNet: Removed LNI 192.168.203.123@tcp [ 2176.227633] Key type .llcrypt unregistered [ 2176.229431] Key type ._llcrypt unregistered [ 2188.439778] Key type ._llcrypt registered [ 2188.441565] Key type .llcrypt registered [ 2188.492953] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2188.971363] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2188.978887] alg: No test for adler32 (adler32-zlib) [ 2189.868541] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 2189.982430] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 2189.985071] LNet: Accept secure, port 988 [ 2191.583165] Key type lgssc registered [ 2192.101445] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2195.685383] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2195.699130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2196.916239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2198.597655] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2202.551778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2202.765993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2204.453317] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2206.915330] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2208.461430] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2211.554988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2211.748867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2213.956945] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2216.110337] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2221.994195] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 2228.872310] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2229.850200] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2232.291234] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2232.291677] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2232.292501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2232.297045] Lustre: Skipped 1 previous similar message [ 2237.168782] LustreError: 118561:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2237.246875] Lustre: server umount lustre-OST0000 complete [ 2238.944606] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2238.948152] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2238.954122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2238.957258] Lustre: Skipped 1 previous similar message [ 2242.528871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2242.528978] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2247.648086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2247.650175] Lustre: Skipped 2 previous similar messages [ 2251.745166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2251.748017] Lustre: Skipped 1 previous similar message [ 2253.279108] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2253.315221] LustreError: 118766:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2253.317711] LustreError: 118766:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2253.346584] Lustre: server umount lustre-MDT0000 complete [ 2254.715271] LustreError: 118413:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753442692 with bad export cookie 14205923637179781315 [ 2254.718858] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2254.719993] LustreError: 118413:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2254.779506] LustreError: 118970:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2254.783156] LustreError: 118970:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2254.866828] Lustre: server umount lustre-MDT0001 complete [ 2260.020017] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 2261.202919] Key type lgssc unregistered [ 2261.343555] LNet: 119497:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2262.377458] LNet: Removed LNI 192.168.203.123@tcp [ 2262.747756] Key type .llcrypt unregistered [ 2262.749359] Key type ._llcrypt unregistered [ 2270.695513] Key type ._llcrypt registered [ 2270.697069] Key type .llcrypt registered [ 2270.746172] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2271.143102] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2271.182228] alg: No test for adler32 (adler32-zlib) [ 2272.047865] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 2272.139739] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 2272.141800] LNet: Accept secure, port 988 [ 2273.727147] Key type lgssc registered [ 2274.130383] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2277.121864] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2277.132207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2278.321323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2279.721086] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2282.822808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2282.990534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2284.511226] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2286.466797] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2287.767448] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2290.355222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2290.498779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2292.574813] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2294.717692] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2302.823058] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 2313.185978] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2313.187261] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2313.188522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2313.189483] LustreError: Skipped 1 previous similar message [ 2313.193704] Lustre: Skipped 1 previous similar message [ 2314.607166] LustreError: 122913:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2314.675425] Lustre: server umount lustre-OST0000 complete [ 2318.305218] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2318.305944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2318.310348] Lustre: Skipped 1 previous similar message [ 2318.312828] Lustre: Skipped 1 previous similar message [ 2322.431700] LustreError: 123116:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2322.433842] LustreError: 123116:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2322.457371] Lustre: server umount lustre-MDT0000 complete [ 2323.424290] LustreError: 120669:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2323.433036] LustreError: 120669:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 2323.968629] LustreError: 120649:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753442761 with bad export cookie 10974873891848491918 [ 2323.972954] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2323.973575] LustreError: 120649:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2324.039836] LustreError: 123319:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2324.042467] LustreError: 123319:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2324.136312] Lustre: server umount lustre-MDT0001 complete [ 2329.221920] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 2330.443812] Key type lgssc unregistered [ 2330.597433] LNet: 123845:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2331.621357] LNet: Removed LNI 192.168.203.123@tcp [ 2331.978379] Key type .llcrypt unregistered [ 2331.980047] Key type ._llcrypt unregistered [ 2341.904393] Key type ._llcrypt registered [ 2341.905726] Key type .llcrypt registered [ 2341.951578] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2342.328404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2342.373592] alg: No test for adler32 (adler32-zlib) [ 2343.244976] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 2343.355190] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 2343.357966] LNet: Accept secure, port 988 [ 2344.951131] Key type lgssc registered [ 2345.347937] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2348.347238] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2348.356881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2349.558743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2351.076639] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2354.406711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2354.557285] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2356.046053] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2358.133332] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2359.510149] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2362.198149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2362.360366] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2364.792244] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2367.069333] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2374.631086] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:163 to 0x240000401:193) [ 2381.423974] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2382.874358] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2382.877664] Lustre: Skipped 2 previous similar messages [ 2384.486547] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2384.488895] Lustre: Skipped 2 previous similar messages [ 2387.453839] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2387.466947] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2395.937196] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.23@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2395.942184] Lustre: Skipped 2 previous similar messages [ 2402.299705] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.23@tcp 192.168.0.[2,10]@tcp [ 2404.193138] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.23/24@tcp in log lustre-MDT0000 [ 2404.196392] Lustre: Skipped 2 previous similar messages [ 2411.003164] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.23/24@tcp [ 2411.005377] Lustre: Skipped 1 previous similar message [ 2415.585384] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2415.586296] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2415.588312] LustreError: Skipped 1 previous similar message [ 2415.588913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2415.593974] Lustre: Skipped 1 previous similar message [ 2415.599444] Lustre: Skipped 1 previous similar message [ 2418.478097] LustreError: 129045:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2418.536746] Lustre: server umount lustre-OST0000 complete [ 2420.704802] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2420.705531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2420.710178] Lustre: Skipped 1 previous similar message [ 2425.824325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2425.827311] Lustre: Skipped 2 previous similar messages [ 2425.983547] LustreError: 129246:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2425.986486] LustreError: 129246:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2426.017888] Lustre: server umount lustre-MDT0000 complete [ 2427.382345] LustreError: 126812:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753442864 with bad export cookie 14027438441620995247 [ 2427.386867] LustreError: 126812:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2427.387056] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2427.446674] LustreError: 129450:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2427.448962] LustreError: 129450:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2427.533600] Lustre: server umount lustre-MDT0001 complete [ 2432.488376] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 2433.611556] Key type lgssc unregistered [ 2433.744404] LNet: 129975:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2434.790172] LNet: Removed LNI 192.168.203.123@tcp [ 2435.164625] Key type .llcrypt unregistered [ 2435.166376] Key type ._llcrypt unregistered [ 2447.439046] Key type ._llcrypt registered [ 2447.440297] Key type .llcrypt registered [ 2447.488787] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2447.900353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2447.963524] alg: No test for adler32 (adler32-zlib) [ 2448.871533] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 2448.983166] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 2448.986704] LNet: Accept secure, port 988 [ 2450.583193] Key type lgssc registered [ 2451.009360] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2453.975354] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2453.984865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2455.165456] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2455.170352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2455.176153] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.23/24@tcp [ 2456.631669] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2459.962625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2460.099247] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2460.111204] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.23/24@tcp [ 2460.131906] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2461.684832] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2463.728330] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2465.208951] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2467.963366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2468.123045] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2470.439563] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2472.805990] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2480.425695] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:201 to 0x240000401:225) [ 2490.848473] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2490.849750] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2490.852262] LustreError: Skipped 1 previous similar message [ 2490.852911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2490.857341] Lustre: Skipped 1 previous similar message [ 2492.720144] LustreError: 133794:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2492.799240] Lustre: server umount lustre-OST0000 complete [ 2495.968914] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2495.969516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2495.974279] Lustre: Skipped 1 previous similar message [ 2495.976674] Lustre: Skipped 1 previous similar message [ 2495.981514] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2500.550388] LustreError: 133995:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2500.552668] LustreError: 133995:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2500.588107] Lustre: server umount lustre-MDT0000 complete [ 2501.088062] LustreError: 132202:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2501.095158] LustreError: 132202:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 2502.258914] LustreError: 131471:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753442939 with bad export cookie 5810624323786319202 [ 2502.259577] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2502.264650] LustreError: 131471:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2502.337599] LustreError: 134196:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2502.339744] LustreError: 134196:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2502.437397] Lustre: server umount lustre-MDT0001 complete [ 2507.546837] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 2508.733431] Key type lgssc unregistered [ 2508.890415] LNet: 134722:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2509.925344] LNet: Removed LNI 192.168.203.123@tcp [ 2510.267721] Key type .llcrypt unregistered [ 2510.269106] Key type ._llcrypt unregistered [ 2520.542163] Key type ._llcrypt registered [ 2520.543907] Key type .llcrypt registered [ 2520.593581] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 2527.208938] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2527.546794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2527.567151] alg: No test for adler32 (adler32-zlib) [ 2528.450991] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 2528.556816] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 2528.558792] LNet: Accept secure, port 988 [ 2530.151151] Key type lgssc registered [ 2530.579182] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2534.303903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2537.116815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2539.764108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2542.391119] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2545.190961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2545.210560] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2545.218775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2546.318415] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2546.335176] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2546.378453] Lustre: lustre-MDT0000: new disk, initializing [ 2546.411350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2546.422858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2547.920493] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2552.902438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2552.926357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2552.964466] Lustre: 138052:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2552.994863] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2552.997523] Lustre: Skipped 1 previous similar message [ 2553.036397] Lustre: lustre-MDT0001: new disk, initializing [ 2553.078063] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2553.098587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2553.103655] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2554.600788] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2558.233839] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2559.807447] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2562.605461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2562.637990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2562.758836] Lustre: lustre-OST0000: new disk, initializing [ 2562.761779] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2562.798656] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2565.249072] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2567.170798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2567.175460] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2567.217061] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2568.927824] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2570.544251] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2570.628080] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2572.161225] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2572.249168] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2577.377997] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2577.378649] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2577.381724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2577.387739] Lustre: Skipped 1 previous similar message [ 2579.118104] LustreError: 140962:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2579.188715] Lustre: server umount lustre-OST0000 complete [ 2582.497036] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2582.497969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2582.507288] Lustre: Skipped 1 previous similar message [ 2582.511628] Lustre: Skipped 2 previous similar messages [ 2587.616116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2587.618790] Lustre: Skipped 1 previous similar message [ 2592.736253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2592.738258] Lustre: Skipped 1 previous similar message [ 2595.295599] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2595.332781] LustreError: 141164:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2595.334713] LustreError: 141164:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2595.375631] Lustre: server umount lustre-MDT0000 complete [ 2596.783402] LustreError: 138827:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753443034 with bad export cookie 1768035445176907461 [ 2596.786669] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2596.787945] LustreError: 138827:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2596.862500] LustreError: 141367:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2596.865806] LustreError: 141367:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2596.962972] Lustre: server umount lustre-MDT0001 complete [ 2600.353375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2600.573232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2602.277095] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2606.048718] LustreError: 141842:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2606.056448] LustreError: 141842:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 2606.267683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2606.451988] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2608.460168] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2610.742511] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2612.483259] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2615.692289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2615.857047] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2618.349573] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2620.685504] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2622.628080] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2626.944729] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 2629.733889] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2632.134637] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 2 sec [ 2640.733065] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2640.834838] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2640.962871] Lustre: lustre-OST0001: new disk, initializing [ 2640.973784] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2641.097660] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2647.389697] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2649.713111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2649.717673] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2649.818771] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2655.130092] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2655.341800] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2659.792277] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2660.068647] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2663.642931] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8b81115aa800.ost_server_uuid 50 [ 2665.166668] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8b81115aa800.ost_server_uuid in FULL state after 0 sec [ 2672.641701] LustreError: 146081:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 2672.644478] LustreError: 146081:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2672.738173] Lustre: server umount lustre-OST0001 complete [ 2675.680969] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2675.683676] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2675.687018] LustreError: Skipped 2 previous similar messages [ 2675.688286] LustreError: 143428:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2675.690362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2675.724599] Lustre: Skipped 3 previous similar messages [ 2680.805753] LustreError: 143428:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2680.814755] LustreError: 143428:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 2681.729136] LustreError: 146282:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2681.732185] LustreError: 146282:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2681.879438] Lustre: server umount lustre-OST0000 complete [ 2685.921123] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2685.929608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2685.932051] Lustre: Skipped 3 previous similar messages [ 2690.343494] LustreError: 146484:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 2690.345701] LustreError: 146484:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2690.404395] Lustre: server umount lustre-MDT0000 complete [ 2691.042556] LustreError: 141837:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2691.054335] LustreError: 141837:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 2692.263251] LustreError: 141824:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753443129 with bad export cookie 1768035445176908343 [ 2692.265213] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2692.271597] LustreError: 141824:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2692.524543] Lustre: server umount lustre-MDT0001 complete [ 2703.146077] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 2704.573886] Key type lgssc unregistered [ 2704.738608] LNet: 147668:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2705.765703] LNet: Removed LNI 192.168.203.123@tcp [ 2706.189817] Key type .llcrypt unregistered [ 2706.191586] Key type ._llcrypt unregistered [ 2723.411195] Key type ._llcrypt registered [ 2723.412756] Key type .llcrypt registered [ 2723.471757] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 2730.951129] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2731.339257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2731.389699] alg: No test for adler32 (adler32-zlib) [ 2732.270897] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 2732.368403] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 2732.370982] LNet: Accept secure, port 988 [ 2733.959755] Key type lgssc registered [ 2734.452442] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2738.662371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2741.996414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2745.246702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2748.927333] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2752.490206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2752.516694] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2752.526308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2753.634896] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2753.651245] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2753.694306] Lustre: lustre-MDT0000: new disk, initializing [ 2753.751920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2753.770648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2755.840943] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2761.434084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2761.469406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2761.514693] Lustre: 151574:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2761.542560] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2761.545027] Lustre: Skipped 1 previous similar message [ 2761.583659] Lustre: lustre-MDT0001: new disk, initializing [ 2761.630472] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2761.655354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2761.659663] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2763.227072] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2767.555817] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2769.749253] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2773.890358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2773.938144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2774.153506] Lustre: lustre-OST0000: new disk, initializing [ 2774.158207] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2774.218392] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2776.982302] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2781.169411] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2781.725106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2781.729620] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2781.815549] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2783.262554] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2783.384312] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2785.286522] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2785.401265] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2786.786837] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2786.787278] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2786.787636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2786.792591] Lustre: Skipped 1 previous similar message [ 2788.397441] LustreError: 154480:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2788.475109] Lustre: server umount lustre-OST0000 complete [ 2791.905626] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2791.911125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2791.913170] Lustre: Skipped 1 previous similar message [ 2791.915122] Lustre: Skipped 1 previous similar message [ 2796.558164] LustreError: 154680:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2796.561182] LustreError: 154680:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2796.631192] Lustre: server umount lustre-MDT0000 complete [ 2798.724802] LustreError: 152347:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753443236 with bad export cookie 11996298277546289417 [ 2798.730270] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2798.733473] LustreError: 152347:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2798.820373] LustreError: 154881:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2798.824130] LustreError: 154881:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2798.970887] Lustre: server umount lustre-MDT0001 complete [ 2803.703264] Lustre: DEBUG MARKER: == conf-sanity test 47a: server restart does not lose lru_resize ========================================================== 07:34:00 (1753443240) [ 2811.896894] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 2816.708494] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2821.759640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2825.907937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2829.674667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2833.480342] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2838.023309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2838.101655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2838.257407] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2838.275508] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2838.328498] Lustre: lustre-MDT0000: new disk, initializing [ 2838.386593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2838.402947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2840.518766] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2846.698679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2846.735459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2846.786089] Lustre: 157857:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2846.826238] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2846.829756] Lustre: Skipped 1 previous similar message [ 2846.877032] Lustre: lustre-MDT0001: new disk, initializing [ 2846.960157] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2846.981609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2846.993051] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2849.032403] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2853.713830] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2855.675780] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2859.440747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2859.484831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2859.629613] Lustre: lustre-OST0000: new disk, initializing [ 2859.636261] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2859.695546] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2862.878428] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2863.603299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2863.607949] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2863.669154] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2867.806644] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2870.629105] Lustre: Failing over lustre-OST0000 [ 2870.659186] LustreError: 160277:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2870.662136] LustreError: 160277:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2870.743724] Lustre: server umount lustre-OST0000 complete [ 2885.368121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2885.522153] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2885.529909] Lustre: Skipped 1 previous similar message [ 2885.538981] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2885.551640] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2887.211235] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2887.920635] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2889.632074] Lustre: 149795:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1753443311/real 1753443311] req@00000000b572963a x1838618424437504/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1753443327 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2890.052963] Lustre: Failing over lustre-MDT0000 [ 2890.131854] LustreError: 161242:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2890.134039] LustreError: 161242:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2890.179511] Lustre: server umount lustre-MDT0000 complete [ 2890.722165] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2890.725156] LustreError: 157863:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2890.732255] Lustre: Skipped 2 previous similar messages [ 2890.745953] LustreError: 157863:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 5 previous similar messages [ 2892.696555] LustreError: 160393:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2894.815782] Lustre: 149796:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1753443316/real 1753443316] req@00000000b517f1e1 x1838618424438784/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1753443332 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2894.837040] Lustre: 149796:0:(client.c:2453:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 2895.840153] LustreError: 157867:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2895.848342] LustreError: 157867:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 2897.870537] LustreError: 157863:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2899.935354] Lustre: 149794:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1753443321/real 1753443321] req@00000000f09408fe x1838618424439936/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1753443337 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2902.990141] LustreError: 157863:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2903.005756] LustreError: 157863:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 2905.165068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2905.257995] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2905.266255] Lustre: 157859:0:(mgc_request.c:1876:mgc_process_log()) MGC192.168.203.123@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2905.388926] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2905.392559] LustreError: Skipped 1 previous similar message [ 2905.464409] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2907.767181] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2908.101085] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2910.694092] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 2910.706352] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2910.739514] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2915.809682] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2915.814368] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2915.822183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2915.825466] Lustre: Skipped 2 previous similar messages [ 2919.085176] LustreError: 162121:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2919.088113] LustreError: 162121:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2919.172979] Lustre: server umount lustre-OST0000 complete [ 2926.048595] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2926.054160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2926.059152] Lustre: Skipped 2 previous similar messages [ 2927.608240] Lustre: server umount lustre-MDT0000 complete [ 2929.746927] LustreError: 157847:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753443367 with bad export cookie 11996298277546291489 [ 2929.749568] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2929.753813] LustreError: 157847:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2930.000744] Lustre: server umount lustre-MDT0001 complete [ 2936.936813] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 2938.613355] Key type lgssc unregistered [ 2938.798561] LNet: 163052:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2939.813422] LNet: Removed LNI 192.168.203.123@tcp [ 2940.248606] Key type .llcrypt unregistered [ 2940.252378] Key type ._llcrypt unregistered [ 2972.481616] Key type ._llcrypt registered [ 2972.482841] Key type .llcrypt registered [ 2972.615340] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2973.665924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2973.700191] alg: No test for adler32 (adler32-zlib) [ 2975.542760] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 2975.951026] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 2975.953410] LNet: Accept secure, port 988 [ 2977.658487] Key type lgssc registered [ 2979.426905] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2990.935891] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2990.962450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2992.974444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2999.568392] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3003.364841] LustreError: 164540:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3003.398963] LustreError: 164540:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3008.501931] LustreError: 164540:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3008.848433] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3013.603977] LustreError: 164540:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3014.087489] LustreError: 164569:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3014.097191] LustreError: 164569:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3014.100919] LustreError: 164569:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 3014.265118] LustreError: 165087:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3014.721971] Lustre: server umount lustre-MDT0000 complete [ 3033.530537] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 3043.146264] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 3052.443896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3057.801708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3063.414244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3069.987451] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3076.083229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3076.144423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3076.296429] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3076.318275] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3076.364783] Lustre: lustre-MDT0000: new disk, initializing [ 3076.431174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3076.446979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3079.816324] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3090.036939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3090.092936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3090.202751] Lustre: 167717:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3090.294420] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3090.307975] Lustre: Skipped 1 previous similar message [ 3090.406677] Lustre: lustre-MDT0001: new disk, initializing [ 3090.520939] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3090.563621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3090.580502] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3094.002119] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3100.608572] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3103.634807] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3109.900819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3109.972832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3110.213536] Lustre: lustre-OST0000: new disk, initializing [ 3110.217057] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3110.349711] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3115.790886] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3118.119875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3118.137240] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3118.249237] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3122.791436] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3126.657799] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 3128.677319] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 07:39:24 (1753443564) [ 3343.841595] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3343.842228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3343.855163] Lustre: Skipped 2 previous similar messages [ 3343.857489] Lustre: Skipped 1 previous similar message [ 3348.448354] LustreError: 171218:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3348.451507] LustreError: 171218:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3348.589805] Lustre: server umount lustre-MDT0000 complete [ 3348.959985] LustreError: 167722:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3348.976135] LustreError: 167722:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3352.223165] LustreError: 169450:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753443789 with bad export cookie 18222351049535435813 [ 3352.229634] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3352.240154] LustreError: 169450:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3354.081382] LustreError: 171107:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3354.086600] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3354.150222] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3354.160031] Lustre: Skipped 1 previous similar message [ 3357.089582] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3358.486716] LustreError: 171422:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3358.566844] LustreError: 171422:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3359.283955] Lustre: server umount lustre-MDT0001 complete [ 3370.175041] LustreError: 171623:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3370.178066] LustreError: 171623:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3370.409193] Lustre: server umount lustre-OST0000 complete [ 3379.957570] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 3390.160979] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 3399.452763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3405.270349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3410.180182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3415.086034] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3420.545251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3420.600814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3420.755224] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3420.785781] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3420.859117] Lustre: lustre-MDT0000: new disk, initializing [ 3420.940932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3420.968067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3424.694423] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3433.806926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3433.876705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3433.930911] Lustre: 173948:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3434.012950] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3434.020583] Lustre: Skipped 1 previous similar message [ 3434.148360] Lustre: lustre-MDT0001: new disk, initializing [ 3434.270709] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3434.325192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3434.338035] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3437.498694] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3444.150752] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3448.371057] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3455.636212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3455.698639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3455.976147] Lustre: lustre-OST0000: new disk, initializing [ 3455.978703] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3456.066508] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3461.521449] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3464.258935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3464.273047] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3464.442165] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3468.630513] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3472.632868] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3472.838864] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3476.710071] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3476.902268] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3480.039356] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3480.039693] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3480.050755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3484.600694] LustreError: 176860:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3484.613363] LustreError: 176860:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3484.748232] Lustre: server umount lustre-OST0000 complete [ 3490.276372] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3490.281268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3490.291245] Lustre: Skipped 2 previous similar messages [ 3490.299501] Lustre: Skipped 2 previous similar messages [ 3493.799044] LustreError: 177062:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3493.815469] LustreError: 177062:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3493.896129] Lustre: server umount lustre-MDT0000 complete [ 3495.393811] LustreError: 173956:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3495.406165] LustreError: 173956:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3496.890791] LustreError: 173942:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753443934 with bad export cookie 18222351049535594643 [ 3496.898760] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3497.192141] Lustre: server umount lustre-MDT0001 complete [ 3504.476508] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 07:45:40 (1753443940) [ 3518.333651] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 3525.387736] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 3533.108254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3538.458739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3544.188879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3549.763471] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3555.027354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3555.084991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3555.242866] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3555.267483] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3555.366399] Lustre: lustre-MDT0000: new disk, initializing [ 3555.488525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3555.504584] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3558.870516] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3568.947338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3569.011497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3569.089383] Lustre: 180243:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3569.323630] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3573.161884] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3581.170258] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3584.940620] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3591.535590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3591.599406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3591.815941] Lustre: lustre-OST0000: new disk, initializing [ 3591.817884] Lustre: Skipped 1 previous similar message [ 3591.820622] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3591.828181] Lustre: Skipped 2 previous similar messages [ 3596.854559] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3601.959728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3601.963865] Lustre: Skipped 1 previous similar message [ 3601.974648] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3602.032856] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3603.051205] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3612.133260] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3612.139681] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3612.142179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3612.147830] Lustre: Skipped 1 previous similar message [ 3612.154965] Lustre: Skipped 1 previous similar message [ 3614.326722] LustreError: 182772:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3614.332174] LustreError: 182772:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 3614.458736] Lustre: server umount lustre-OST0000 complete [ 3623.180943] Lustre: server umount lustre-MDT0000 complete [ 3626.017481] LustreError: 180235:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753444063 with bad export cookie 18222351049535595525 [ 3626.023772] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3626.031243] LustreError: 180235:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 3626.328450] Lustre: server umount lustre-MDT0001 complete [ 3632.423549] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 07:47:48 (1753444068) [ 3643.001623] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 3648.730952] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 3655.842261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3660.581586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3665.402054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3670.108961] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3675.253801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3675.310748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3675.511176] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3675.556429] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3675.648270] Lustre: lustre-MDT0000: new disk, initializing [ 3675.726942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3675.730129] Lustre: Skipped 2 previous similar messages [ 3675.746858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3679.058815] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3688.230040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3688.285163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3688.368220] Lustre: 186150:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3688.597471] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3691.997870] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3698.493333] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3701.696875] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3707.783409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3707.842788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3713.259777] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3714.234539] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3719.711639] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3729.887909] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3729.891700] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3729.900295] LustreError: Skipped 1 previous similar message [ 3729.912613] Lustre: Skipped 2 previous similar messages [ 3729.914384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3729.924851] Lustre: Skipped 5 previous similar messages [ 3731.515292] LustreError: 188678:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3731.521577] LustreError: 188678:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3731.656524] Lustre: server umount lustre-OST0000 complete [ 3734.501917] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3740.489622] Lustre: server umount lustre-MDT0000 complete [ 3743.309716] LustreError: 186142:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753444180 with bad export cookie 18222351049535596610 [ 3743.317213] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3743.321745] LustreError: 186142:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3743.566768] Lustre: server umount lustre-MDT0001 complete [ 3751.803953] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 3753.812165] Key type lgssc unregistered [ 3754.078537] LNet: 189608:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3755.110132] LNet: Removed LNI 192.168.203.123@tcp [ 3755.834760] Key type .llcrypt unregistered [ 3755.837212] Key type ._llcrypt unregistered [ 3772.116190] Key type ._llcrypt registered [ 3772.117707] Key type .llcrypt registered [ 3772.216725] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 3772.865564] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3772.944713] alg: No test for adler32 (adler32-zlib) [ 3773.997502] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 3774.169656] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 3774.171717] LNet: Accept secure, port 988 [ 3775.799162] Key type lgssc registered [ 3776.722697] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3782.462369] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3782.495972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3783.819745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3786.820819] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3789.281341] LustreError: 190972:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3789.293170] LustreError: 190972:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3793.542712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3793.858525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3797.034123] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3801.894349] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3804.853971] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3810.078402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3810.211774] LustreError: 192557:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3810.326816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3814.468539] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3815.406454] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3819.186854] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3839.975248] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3839.984351] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3839.984427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3840.010937] Lustre: Skipped 1 previous similar message [ 3842.481131] LustreError: 193219:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3842.605401] Lustre: server umount lustre-OST0000 complete [ 3850.211050] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3850.211325] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3850.223158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3850.235670] Lustre: Skipped 2 previous similar messages [ 3851.884605] LustreError: 193421:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3851.895275] LustreError: 193421:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3851.995501] Lustre: server umount lustre-MDT0000 complete [ 3855.116045] LustreError: 191684:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753444292 with bad export cookie 1712029672084274615 [ 3855.129080] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3855.261876] LustreError: 193625:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3855.267696] LustreError: 193625:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3855.464898] Lustre: server umount lustre-MDT0001 complete [ 3865.081914] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 3867.410558] Key type lgssc unregistered [ 3867.671869] LNet: 194151:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3868.711680] LNet: Removed LNI 192.168.203.123@tcp [ 3869.326091] Key type .llcrypt unregistered [ 3869.328703] Key type ._llcrypt unregistered [ 3888.441961] Key type ._llcrypt registered [ 3888.444700] Key type .llcrypt registered [ 3888.539154] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 3889.276358] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3889.302490] alg: No test for adler32 (adler32-zlib) [ 3890.324585] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 3890.530282] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 3890.534140] LNet: Accept secure, port 988 [ 3892.191158] Key type lgssc registered [ 3893.085195] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3898.747544] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3898.769509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3900.107544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3902.778676] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3908.494629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3908.819701] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3911.679391] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3916.095296] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3918.958950] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3924.787948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3925.104962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3926.124277] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 3929.764434] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3934.055976] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3939.819357] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3939.826730] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3939.829179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3939.843413] Lustre: Skipped 1 previous similar message [ 3940.838023] Lustre: lustre-OST0000: Not available for connect from 192.168.203.23@tcp (stopping) [ 3940.840440] Lustre: Skipped 1 previous similar message [ 3944.932686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3945.136540] LustreError: 197740:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3945.261767] Lustre: server umount lustre-OST0000 complete [ 3949.411132] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8b810646a000.ost_server_uuid 50 [ 3950.476798] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 3962.182187] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3962.189266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3962.192138] Lustre: Skipped 1 previous similar message [ 3962.201576] Lustre: Skipped 1 previous similar message [ 3967.459803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3967.473574] Lustre: Skipped 2 previous similar messages [ 3968.349899] LustreError: 198111:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3968.353657] LustreError: 198111:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3968.455615] Lustre: server umount lustre-MDT0000 complete [ 3970.531172] LustreError: 195516:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3970.553834] LustreError: 195516:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 3971.291971] LustreError: 195498:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753444408 with bad export cookie 2901692734249438375 [ 3971.295289] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3971.303179] LustreError: 195498:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3971.448962] LustreError: 198313:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3971.456212] LustreError: 198313:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3971.624916] Lustre: server umount lustre-MDT0001 complete [ 3978.419344] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 07:53:34 (1753444414) [ 3983.635434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3984.023362] LustreError: 198985:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3984.111371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3987.682622] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3989.472757] LustreError: 198985:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3993.579189] LustreError: 198985:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3993.965755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3994.233982] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3997.129955] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4001.388564] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4003.987439] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4008.412725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4008.658071] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4009.709330] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4012.697442] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4016.437424] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4020.129250] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4020.178190] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4020.279944] Lustre: lustre-OST0001: new disk, initializing [ 4020.285357] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4023.821271] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4028.454456] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4030.432681] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4030.438778] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4030.446353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4030.470720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4030.478545] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4030.540532] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4036.210105] LustreError: 202150:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4036.212698] LustreError: 202150:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4036.241337] Lustre: server umount lustre-OST0000 complete [ 4039.119933] LustreError: 202159:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4039.313305] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4039.423296] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4041.679748] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4041.827714] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4049.364772] LustreError: 202160:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4049.399905] LustreError: 202160:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 5 previous similar messages [ 4056.038249] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4056.041802] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4056.047593] LustreError: Skipped 1 previous similar message [ 4056.047667] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4056.047671] Lustre: Skipped 4 previous similar messages [ 4056.054618] Lustre: Skipped 2 previous similar messages [ 4060.266919] LustreError: 202872:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 4060.286311] LustreError: 202872:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4060.431146] Lustre: server umount lustre-OST0001 complete [ 4065.760109] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4065.763334] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4069.779379] LustreError: 203074:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4069.784370] LustreError: 203074:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4069.857725] Lustre: server umount lustre-MDT0000 complete [ 4071.394079] LustreError: 200082:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4071.403620] LustreError: 200082:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 5 previous similar messages [ 4072.313400] LustreError: 198966:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753444509 with bad export cookie 2901692734249439446 [ 4072.313677] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4072.323590] LustreError: 198966:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4072.567221] Lustre: server umount lustre-MDT0001 complete [ 4076.216756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4078.620712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4081.223351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4083.603907] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4087.788236] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 07:55:24 (1753444524) [ 4091.567901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4091.756376] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4091.788210] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4091.935243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4091.942752] Lustre: Skipped 1 previous similar message [ 4094.419365] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4100.582527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4100.662698] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4100.692370] Lustre: 204767:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4103.385739] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4107.082961] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4109.521626] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4114.163976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4114.313956] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4115.808671] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4118.798346] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4122.989810] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4128.366879] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4128.438037] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4128.581932] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4128.584698] Lustre: Skipped 2 previous similar messages [ 4132.327307] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4135.423253] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4138.496711] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4138.501794] Lustre: Skipped 1 previous similar message [ 4138.504735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4138.514855] Lustre: Skipped 4 previous similar messages [ 4143.413085] LustreError: 207928:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4143.420746] LustreError: 207928:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4143.447942] Lustre: server umount lustre-OST0000 complete [ 4143.587830] LustreError: 206385:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4143.594160] LustreError: 206385:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 4158.948186] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4158.955426] Lustre: Skipped 2 previous similar messages [ 4158.958822] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4161.701523] Lustre: server umount lustre-OST0001 complete [ 4167.648059] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4167.650877] LustreError: Skipped 1 previous similar message [ 4178.399705] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4178.464139] LustreError: 208341:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4178.468098] LustreError: 208341:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4178.532909] Lustre: server umount lustre-MDT0000 complete [ 4180.758379] LustreError: 204758:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753444618 with bad export cookie 2901692734249440818 [ 4180.762802] LustreError: 204758:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4180.762818] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4180.964700] Lustre: server umount lustre-MDT0001 complete [ 4184.221940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4186.464248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4188.621608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4190.822058] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4195.739187] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 07:57:11 (1753444631) [ 4205.775427] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 4212.398129] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 4218.965440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4223.911235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4228.242074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4232.526616] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4237.078486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4237.129507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4237.276525] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4237.310290] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4237.357021] Lustre: lustre-MDT0000: new disk, initializing [ 4237.433191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4237.449743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4240.246948] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4248.778342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4248.852620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4248.911770] Lustre: 212351:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4248.976730] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4248.979451] Lustre: Skipped 1 previous similar message [ 4249.031084] Lustre: lustre-MDT0001: new disk, initializing [ 4249.152360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4249.158423] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4251.808563] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4257.337795] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4259.726444] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4263.753938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4263.801914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4263.955040] Lustre: lustre-OST0000: new disk, initializing [ 4263.960115] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4267.451681] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4271.647405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4271.652982] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4271.725725] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4272.192810] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4274.672920] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4274.806845] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4276.915846] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4277.058408] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4281.827958] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4281.832593] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4281.832797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4281.832801] Lustre: Skipped 9 previous similar messages [ 4281.837504] Lustre: Skipped 3 previous similar messages [ 4281.849818] LustreError: Skipped 1 previous similar message [ 4284.143488] LustreError: 215258:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4284.146975] LustreError: 215258:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4284.246537] Lustre: server umount lustre-OST0000 complete [ 4292.804939] Lustre: server umount lustre-MDT0000 complete [ 4295.068259] LustreError: 212341:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753444732 with bad export cookie 2901692734249442351 [ 4295.068905] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4295.074602] LustreError: 212341:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4295.300949] Lustre: server umount lustre-MDT0001 complete [ 4300.569977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4300.830890] LustreError: 216138:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4300.839244] LustreError: 216138:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 10 previous similar messages [ 4303.152375] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4308.467466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4310.571486] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4313.808209] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4316.495699] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4321.231930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4324.932580] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4328.560333] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4330.867196] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4330.980354] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4333.313520] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4334.459140] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 1 sec [ 4336.097139] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4341.676472] Lustre: server umount lustre-OST0000 complete [ 4345.640161] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4348.361893] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4374.568286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4374.775502] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4374.779690] Lustre: Skipped 5 previous similar messages [ 4378.208917] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4378.856149] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4378.864581] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to (at 0@lo) [ 4382.291889] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4389.094830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4389.101856] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.123@tcp (at 0@lo) [ 4394.254741] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 10 sec [ 4399.585702] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4399.600141] Lustre: Skipped 5 previous similar messages [ 4402.459195] Lustre: server umount lustre-OST0000 complete [ 4406.244208] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4406.247886] LustreError: Skipped 2 previous similar messages [ 4411.232474] Lustre: server umount lustre-MDT0000 complete [ 4413.964400] LustreError: 216118:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753444851 with bad export cookie 2901692734249443233 [ 4413.966857] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4413.969921] LustreError: 216118:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4414.070592] LustreError: 220904:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4414.074804] LustreError: 220904:0:(obd_class.h:479:obd_check_dev()) Skipped 31 previous similar messages [ 4414.225428] Lustre: server umount lustre-MDT0001 complete [ 4420.227660] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 08:00:56 (1753444856) [ 4425.752356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4429.174346] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4431.330939] LustreError: 221584:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4431.343240] LustreError: 221584:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 4434.938784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4437.523612] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4441.056477] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4443.163506] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4447.105772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4451.255588] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4455.301939] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4458.150504] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4458.309218] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4460.919414] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4461.072941] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4465.311122] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4465.371342] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4465.460959] Lustre: lustre-OST0001: new disk, initializing [ 4465.465169] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4468.756568] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4474.026685] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4474.936853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4474.949749] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4475.004551] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4476.726261] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4476.918089] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4479.307307] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4479.434736] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4482.716965] Lustre: server umount lustre-OST0001 complete [ 4486.433690] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4486.561476] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4488.917809] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4489.059047] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4515.088732] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4518.775646] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4520.432732] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4520.441762] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to (at 0@lo) [ 4520.441825] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4522.110394] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4523.200817] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4525.551128] Lustre: lustre-OST0001: Not available for connect from 192.168.203.23@tcp (stopping) [ 4525.553887] Lustre: Skipped 13 previous similar messages [ 4530.512735] Lustre: server umount lustre-OST0001 complete [ 4546.019431] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4546.025995] Lustre: Skipped 6 previous similar messages [ 4550.668842] Lustre: server umount lustre-OST0000 complete [ 4553.189710] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4553.193148] LustreError: Skipped 3 previous similar messages [ 4559.132527] Lustre: server umount lustre-MDT0000 complete [ 4561.593608] LustreError: 224555:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753444999 with bad export cookie 2901692734249444185 [ 4561.595274] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4561.599675] LustreError: 224555:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4567.954752] Lustre: server umount lustre-MDT0001 complete [ 4572.144981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4574.794334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4577.440712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4580.058211] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4584.408848] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 08:03:40 (1753445020) [ 4588.854495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4589.038445] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4589.059589] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4591.801103] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4597.366760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4597.438430] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4597.468987] Lustre: 229487:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4600.270782] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4604.262881] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4606.947946] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4611.435491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4611.586701] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4615.872827] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4620.065468] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4626.918486] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4626.975847] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4630.831383] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4634.174383] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4635.158868] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4637.395304] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4637.522382] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4639.780437] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4639.910774] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4642.242690] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8b81090d1800.ost_server_uuid 50 [ 4643.011348] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8b81090d1800.ost_server_uuid in FULL state after 0 sec [ 4643.568327] Lustre: Permanently deactivating lustre-OST0001 [ 4643.576870] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 4645.056589] Lustre: Permanently reactivating lustre-OST0001 [ 4653.061359] Lustre: server umount lustre-OST0001 complete [ 4668.895137] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4669.003638] Lustre: server umount lustre-OST0000 complete [ 4677.291566] LustreError: 233844:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4677.293874] LustreError: 233844:0:(obd_class.h:479:obd_check_dev()) Skipped 45 previous similar messages [ 4677.368240] Lustre: server umount lustre-MDT0000 complete [ 4679.730402] LustreError: 229477:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753445117 with bad export cookie 2901692734249445466 [ 4679.738767] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4680.009220] Lustre: server umount lustre-MDT0001 complete [ 4683.538156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4685.851755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4688.295743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4690.882563] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4695.382075] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 08:05:31 (1753445131) [ 4695.964418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4699.564773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4699.706176] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4699.721917] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4699.726098] Lustre: Skipped 5 previous similar messages [ 4699.817050] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4699.820360] Lustre: Skipped 9 previous similar messages [ 4701.674701] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4705.664285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4705.714232] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4705.727627] Lustre: 235596:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4707.614479] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4710.664806] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4712.549471] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4716.832793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4716.886493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4717.000768] Lustre: Permanently deactivating lustre-OST0000 [ 4717.007141] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 4720.333270] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4723.870941] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4726.791111] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4728.237548] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4731.897484] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4735.241756] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4736.047289] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 4741.270116] Lustre: Permanently reactivating lustre-OST0000 [ 4741.279503] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 4741.282776] Lustre: Skipped 2 previous similar messages [ 4746.257314] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 4746.260624] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4746.265800] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.203.123@tcp (at 0@lo) [ 4746.269228] Lustre: Skipped 1 previous similar message [ 4746.285034] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4746.294804] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4759.757717] Lustre: server umount lustre-OST0001 complete [ 4761.574212] LustreError: 237205:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4761.581314] LustreError: 237205:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 35 previous similar messages [ 4768.144300] Lustre: server umount lustre-OST0000 complete [ 4776.594357] Lustre: server umount lustre-MDT0000 complete [ 4778.705195] LustreError: 237229:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753445216 with bad export cookie 2901692734249447125 [ 4778.706286] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4778.709476] LustreError: 237229:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4778.950211] Lustre: server umount lustre-MDT0001 complete [ 4785.287217] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 4786.655368] Key type lgssc unregistered [ 4786.821942] LNet: 239979:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4787.877793] LNet: Removed LNI 192.168.203.123@tcp [ 4788.264203] Key type .llcrypt unregistered [ 4788.265227] Key type ._llcrypt unregistered [ 4799.948294] Key type ._llcrypt registered [ 4799.950123] Key type .llcrypt registered [ 4800.000184] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 4800.383925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4800.438963] alg: No test for adler32 (adler32-zlib) [ 4801.321033] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 4801.428241] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 4801.430717] LNet: Accept secure, port 988 [ 4803.031169] Key type lgssc registered [ 4803.451127] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4805.044474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4808.625624] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4808.636177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4809.937239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4811.849995] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4816.152635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4816.194838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4816.253913] Lustre: Found index 1 for lustre-MDT0001, updating log [ 4816.268939] Lustre: 241384:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4816.276891] Lustre: Permanently deactivating lustre-MDT0001 [ 4816.286471] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 4816.449882] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4818.399052] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4818.613685] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 4818.619989] LustreError: 241425:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4818.626406] LustreError: 241425:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4818.632580] LustreError: 241425:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 1, failed: rc = -108 [ 4821.123254] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4822.893637] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4826.486907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4826.683055] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4829.355373] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4832.095565] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4834.851996] LustreError: 242998:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4834.859963] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 4834.861735] LustreError: 242998:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 4835.567912] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4835.687764] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4838.509464] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4840.816816] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 4840.823205] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 4840.823581] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 4841.536889] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4844.458497] Lustre: Permanently reactivating lustre-MDT0001 [ 4844.465774] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 4844.468940] Lustre: Skipped 2 previous similar messages [ 4848.609166] Lustre: lustre-MDT0001: Client e779c163-21ca-43b1-9740-c0b255e3cd3c (at 192.168.203.23@tcp) reconnecting [ 4850.402408] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4851.175426] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4853.368465] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 4853.454759] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 4855.291907] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 4855.401879] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 4856.810481] Lustre: Permanently deactivating lustre-MDT0001 [ 4856.819501] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 4856.823030] Lustre: Skipped 2 previous similar messages [ 4863.512725] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 4866.874476] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 4866.878246] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4866.884933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4871.653599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4871.653873] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4871.656516] Lustre: Skipped 1 previous similar message [ 4871.665204] Lustre: Skipped 2 previous similar messages [ 4873.030912] LustreError: 245468:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 4873.048535] LustreError: 245468:0:(ldlm_resource.c:981:ldlm_resource_complain()) MGS: namespace resource [0x736d61726170:0x3:0x0].0x0 (00000000469e3d01) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 4875.107203] LustreError: 245468:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 4875.109914] LustreError: 245468:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4875.132452] Lustre: server umount lustre-MDT0000 complete [ 4876.663352] LustreError: 245264:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753445314 with bad export cookie 14429523931033996347 [ 4876.665445] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4876.669126] LustreError: 245264:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4876.745475] LustreError: 245670:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 4876.747378] LustreError: 245670:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 4876.767789] LustreError: 243495:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4876.774378] LustreError: 243495:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 4876.844669] Lustre: server umount lustre-MDT0001 complete [ 4884.463645] LustreError: 245870:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 4884.466285] LustreError: 245870:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4884.490169] Lustre: server umount lustre-OST0000 complete [ 4889.489174] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 08:08:46 (1753445326) [ 4894.476078] LustreError: 246875:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4894.478198] LustreError: 246875:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 4894.562546] Lustre: server umount lustre-OST0001 complete [ 4897.542623] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 4902.199116] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 4906.815303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4910.271442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4913.640158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4916.665091] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4919.720843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4919.748539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4919.840128] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4919.844627] Lustre: Skipped 2 previous similar messages [ 4919.859473] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4919.901649] Lustre: lustre-MDT0000: new disk, initializing [ 4919.932210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4919.942750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4921.629177] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4926.849901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4926.883620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4926.927810] Lustre: 249042:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4926.963814] LustreError: 249054:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4926.969894] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4926.975872] Lustre: Skipped 1 previous similar message [ 4927.015773] Lustre: lustre-MDT0001: new disk, initializing [ 4927.078298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4927.084351] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4928.901824] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4933.069093] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4934.988513] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4938.290080] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4938.321158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4938.464102] Lustre: lustre-OST0000: new disk, initializing [ 4938.467874] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4938.513490] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4938.516239] Lustre: Skipped 1 previous similar message [ 4938.877670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4938.886476] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4938.937741] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4941.463765] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4946.045609] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4947.340586] LustreError: 249048:0:(mdt_handler.c:3257:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4952.892950] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4952.938509] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4953.043164] Lustre: lustre-OST0001: new disk, initializing [ 4953.044596] Lustre: lustre-OST0001: Not available for connect from 192.168.203.23@tcp (not set up) [ 4953.049907] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4953.053227] Lustre: Skipped 2 previous similar messages [ 4955.967384] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4957.359142] LustreError: 249048:0:(mdt_handler.c:3257:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4957.372255] LustreError: 249048:0:(mdt_handler.c:3257:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4958.748541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4958.752804] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4958.791041] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4960.303826] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4961.283673] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4967.471150] LustreError: 249048:0:(mdt_handler.c:3257:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4968.936249] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4968.940166] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4968.947294] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4973.536120] Lustre: lustre-OST0001: Not available for connect from 192.168.203.23@tcp (stopping) [ 4973.540627] Lustre: Skipped 1 previous similar message [ 4983.263970] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4983.289591] LustreError: 252510:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 4983.295075] LustreError: 252510:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4983.317558] Lustre: server umount lustre-OST0001 complete [ 4983.760872] LustreError: 250761:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4989.411047] LustreError: 250762:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4989.421232] LustreError: 250762:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 4999.656027] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4999.660149] LustreError: 250827:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4999.669300] Lustre: Skipped 1 previous similar message [ 4999.670186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4999.680139] LustreError: 250827:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 4999.687215] Lustre: Skipped 6 previous similar messages [ 5002.868120] LustreError: 252735:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5002.871669] LustreError: 252735:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5002.961110] Lustre: server umount lustre-OST0000 complete [ 5008.864034] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5008.866997] LustreError: Skipped 1 previous similar message [ 5008.871253] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5008.877892] Lustre: Skipped 1 previous similar message [ 5011.177522] Lustre: server umount lustre-MDT0000 complete [ 5012.765291] LustreError: 249817:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753445450 with bad export cookie 14429523931033999154 [ 5012.768239] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5012.771963] LustreError: 249817:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5027.295152] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5027.431912] Lustre: server umount lustre-MDT0001 complete [ 5033.816954] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 5035.160221] Key type lgssc unregistered [ 5035.331829] LNet: 253667:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5036.389528] LNet: Removed LNI 192.168.203.123@tcp [ 5036.739790] Key type .llcrypt unregistered [ 5036.741618] Key type ._llcrypt unregistered [ 5052.099906] Key type ._llcrypt registered [ 5052.101227] Key type .llcrypt registered [ 5052.145455] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 5060.118478] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 5060.445324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5060.456912] alg: No test for adler32 (adler32-zlib) [ 5061.325358] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 5061.423511] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 5061.426658] LNet: Accept secure, port 988 [ 5063.015291] Key type lgssc registered [ 5063.571680] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5068.113482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5071.490996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5074.578211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5077.704261] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5081.116818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5081.154715] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5081.162848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5082.292198] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5082.308076] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5082.350313] Lustre: lustre-MDT0000: new disk, initializing [ 5082.391189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5082.407405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5084.448093] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5089.892218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5089.927597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5089.971224] Lustre: 257575:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5090.005173] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5090.008084] Lustre: Skipped 1 previous similar message [ 5090.048648] Lustre: lustre-MDT0001: new disk, initializing [ 5090.100518] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5090.119307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5090.125503] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5091.944324] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5096.313732] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5098.531168] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5102.539516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5102.589618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5102.743318] Lustre: lustre-OST0000: new disk, initializing [ 5102.746110] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5102.791743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5103.038840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5103.047566] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5103.117522] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5105.679523] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5110.433850] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5112.793411] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5112.926262] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5115.261833] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5115.383880] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5118.432773] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5118.433782] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5118.436367] LustreError: Skipped 1 previous similar message [ 5118.438776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5118.442238] Lustre: Skipped 1 previous similar message [ 5122.735101] LustreError: 260481:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5122.827408] Lustre: server umount lustre-OST0000 complete [ 5126.114415] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5126.119385] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5126.135058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5126.139793] Lustre: Skipped 1 previous similar message [ 5128.681832] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5128.684995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5130.828483] LustreError: 260682:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5130.833076] LustreError: 260682:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5130.883400] Lustre: server umount lustre-MDT0000 complete [ 5132.813148] LustreError: 258347:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753445570 with bad export cookie 16327849558509426649 [ 5132.817564] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5132.820617] LustreError: 258347:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5132.912807] LustreError: 260885:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5132.915260] LustreError: 260885:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5133.033373] Lustre: server umount lustre-MDT0001 complete [ 5137.940754] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 08:12:54 (1753445574) [ 5141.668311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5141.983583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5144.105861] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5147.106623] LustreError: 261554:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5147.123244] LustreError: 261554:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 5148.691544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5148.875083] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5150.769919] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5154.011745] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5155.986656] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5159.572591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5159.756673] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5162.579907] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5165.252908] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5215.200884] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5215.203254] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5215.204903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5215.204910] Lustre: Skipped 1 previous similar message [ 5215.221285] Lustre: Skipped 1 previous similar message [ 5220.325662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5220.334403] Lustre: Skipped 2 previous similar messages [ 5221.104123] LustreError: 263859:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5221.107446] LustreError: 263859:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5221.258770] Lustre: server umount lustre-OST0000 complete [ 5223.815159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 5230.458200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5230.644400] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5234.108354] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5237.497494] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5244.968498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5244.980488] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.123@tcp (at 0@lo) [ 5244.985088] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5275.619814] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5275.624651] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5275.625284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5275.631364] Lustre: Skipped 1 previous similar message [ 5275.639751] LustreError: Skipped 1 previous similar message [ 5278.839192] LustreError: 265513:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5278.852871] LustreError: 265513:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5278.985239] Lustre: server umount lustre-OST0000 complete [ 5282.273285] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5285.859936] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5285.866245] Lustre: Skipped 1 previous similar message [ 5287.375025] LustreError: 265716:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5287.380690] LustreError: 265716:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5287.415705] Lustre: server umount lustre-MDT0000 complete [ 5289.549684] LustreError: 261536:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753445727 with bad export cookie 16327849558509427531 [ 5289.552791] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5289.557369] LustreError: 261536:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5289.772045] Lustre: server umount lustre-MDT0001 complete [ 5296.661850] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 5298.166385] Key type lgssc unregistered [ 5298.354872] LNet: 266444:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5299.432144] LNet: Removed LNI 192.168.203.123@tcp [ 5299.855632] Key type .llcrypt unregistered [ 5299.857205] Key type ._llcrypt unregistered [ 5312.595229] Key type ._llcrypt registered [ 5312.596737] Key type .llcrypt registered [ 5312.646747] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 5313.108250] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5313.133750] alg: No test for adler32 (adler32-zlib) [ 5314.025597] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 5314.131105] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 5314.133846] LNet: Accept secure, port 988 [ 5315.727813] Key type lgssc registered [ 5316.141237] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5319.211438] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5319.219834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5320.401666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5322.040695] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5325.656054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5325.827709] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5327.366590] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5329.735567] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5331.486705] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5334.623498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5334.778377] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5337.335477] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5340.018916] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5345.005390] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 5355.489308] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5355.492158] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5355.492542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5355.496328] Lustre: Skipped 1 previous similar message [ 5360.429255] LustreError: 270809:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5360.536163] Lustre: server umount lustre-OST0000 complete [ 5365.731340] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5365.732578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5365.736756] Lustre: Skipped 1 previous similar message [ 5365.739743] Lustre: Skipped 1 previous similar message [ 5368.586337] LustreError: 271011:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5368.593057] LustreError: 271011:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5368.637678] Lustre: server umount lustre-MDT0000 complete [ 5370.401523] LustreError: 267792:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753445807 with bad export cookie 12452889678445413448 [ 5370.403872] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5370.409899] LustreError: 267792:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5370.494364] LustreError: 271212:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5370.497598] LustreError: 271212:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5370.598385] Lustre: server umount lustre-MDT0001 complete [ 5377.006239] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 5378.457613] Key type lgssc unregistered [ 5378.637240] LNet: 271738:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5379.684638] LNet: Removed LNI 192.168.203.123@tcp [ 5380.193127] Key type .llcrypt unregistered [ 5380.195894] Key type ._llcrypt unregistered [ 5389.736278] Key type ._llcrypt registered [ 5389.737992] Key type .llcrypt registered [ 5389.808155] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 5390.298812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5390.356386] alg: No test for adler32 (adler32-zlib) [ 5391.268918] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 5391.393462] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 5391.395622] LNet: Accept secure, port 988 [ 5392.991157] Key type lgssc registered [ 5393.504737] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5396.876250] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5396.887075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5398.107508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5399.809375] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5404.096686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5404.333202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5406.527873] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5409.691365] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5412.020882] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5415.865212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5416.087662] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5419.169066] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5421.924445] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5423.284731] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5438.944170] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5438.946806] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5438.947655] LustreError: Skipped 1 previous similar message [ 5438.951828] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5439.969582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5439.970483] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5439.970609] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5442.484112] LustreError: 275298:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5442.600974] Lustre: server umount lustre-OST0000 complete [ 5445.090116] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5445.094696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5445.095149] Lustre: Skipped 1 previous similar message [ 5445.097476] Lustre: Skipped 1 previous similar message [ 5450.209681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5450.214646] Lustre: Skipped 1 previous similar message [ 5450.440703] LustreError: 275499:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5450.444223] LustreError: 275499:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5450.500475] Lustre: server umount lustre-MDT0000 complete [ 5452.512727] LustreError: 272889:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753445889 with bad export cookie 3505585365008559073 [ 5452.517048] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5452.520053] LustreError: 272889:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5452.598986] LustreError: 275703:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5452.606484] LustreError: 275703:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5452.773618] Lustre: server umount lustre-MDT0001 complete [ 5459.940193] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 5461.595818] Key type lgssc unregistered [ 5461.780745] LNet: 276229:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5462.821457] LNet: Removed LNI 192.168.203.123@tcp [ 5463.246318] Key type .llcrypt unregistered [ 5463.248368] Key type ._llcrypt unregistered [ 5476.545339] Key type ._llcrypt registered [ 5476.547321] Key type .llcrypt registered [ 5476.608465] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 5477.133169] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5477.144512] alg: No test for adler32 (adler32-zlib) [ 5478.025364] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 5478.141599] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 5478.143942] LNet: Accept secure, port 988 [ 5479.735155] Key type lgssc registered [ 5480.234503] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5484.582370] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5484.606053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5485.865175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5488.042747] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5492.820751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5493.029173] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5495.178038] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5498.293880] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5500.379732] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5504.291300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5504.490473] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5507.645518] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5510.631500] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5510.637110] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5533.153403] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5533.154940] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5533.161331] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5533.166988] Lustre: Skipped 1 previous similar message [ 5538.275449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5538.281850] Lustre: Skipped 1 previous similar message [ 5538.360539] LustreError: 280613:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5538.457581] Lustre: server umount lustre-OST0000 complete [ 5543.396158] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5543.397317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5543.402516] Lustre: Skipped 2 previous similar messages [ 5546.696918] LustreError: 280815:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5546.700447] LustreError: 280815:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5546.754920] Lustre: server umount lustre-MDT0000 complete [ 5548.512284] LustreError: 277594:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5548.522677] LustreError: 277594:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 5548.714080] LustreError: 277575:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753445986 with bad export cookie 79057699694442405 [ 5548.716418] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5548.720325] LustreError: 277575:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5548.821138] LustreError: 281017:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5548.823300] LustreError: 281017:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5548.935524] Lustre: server umount lustre-MDT0001 complete [ 5555.959618] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 5557.843205] Key type lgssc unregistered [ 5558.035600] LNet: 281544:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5559.079018] LNet: Removed LNI 192.168.203.123@tcp [ 5559.613155] Key type .llcrypt unregistered [ 5559.619876] Key type ._llcrypt unregistered [ 5570.654906] Key type ._llcrypt registered [ 5570.656637] Key type .llcrypt registered [ 5570.718575] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 5571.145536] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5571.153941] alg: No test for adler32 (adler32-zlib) [ 5572.111880] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 5572.262601] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 5572.267796] LNet: Accept secure, port 988 [ 5573.879163] Key type lgssc registered [ 5574.516926] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5578.866177] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5578.883129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5580.210391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5582.671369] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5588.464375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5588.688286] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5591.175758] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5594.854566] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5597.173589] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5601.211129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5601.413228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5604.665556] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5605.490344] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 5608.157990] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5619.173228] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5619.173648] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5619.174710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5619.183782] Lustre: Skipped 1 previous similar message [ 5620.472208] LustreError: 285123:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5620.571159] Lustre: server umount lustre-OST0000 complete [ 5624.289546] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5624.306975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5624.310425] Lustre: Skipped 1 previous similar message [ 5628.807852] LustreError: 285323:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5628.830751] LustreError: 285323:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5628.915197] Lustre: server umount lustre-MDT0000 complete [ 5629.408750] LustreError: 282739:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5629.416329] LustreError: 282739:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 5630.844077] LustreError: 282693:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753446068 with bad export cookie 15085487469462233356 [ 5630.847077] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5630.854494] LustreError: 282693:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5630.932457] LustreError: 285524:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5630.935351] LustreError: 285524:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5631.063283] Lustre: server umount lustre-MDT0001 complete [ 5637.345192] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 5638.747639] Key type lgssc unregistered [ 5638.899295] LNet: 286051:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5639.973414] LNet: Removed LNI 192.168.203.123@tcp [ 5640.411715] Key type .llcrypt unregistered [ 5640.414147] Key type ._llcrypt unregistered [ 5646.146277] Key type ._llcrypt registered [ 5646.148109] Key type .llcrypt registered [ 5646.200759] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 5654.654070] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 5663.307284] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 5663.819612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5663.832708] alg: No test for adler32 (adler32-zlib) [ 5664.750261] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 5664.869406] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 5664.871742] LNet: Accept secure, port 988 [ 5666.464591] Key type lgssc registered [ 5667.071980] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5672.264604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5676.541490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5679.790535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5682.889448] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5686.632682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5686.660394] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5686.669635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5687.783198] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5687.800213] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5687.842664] Lustre: lustre-MDT0000: new disk, initializing [ 5687.911645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5687.920435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5689.775397] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5695.237628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5695.267224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5695.313138] Lustre: 289530:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5695.365791] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5695.367991] Lustre: Skipped 1 previous similar message [ 5695.413384] Lustre: lustre-MDT0001: new disk, initializing [ 5695.455793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5695.472119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5695.476657] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5697.124300] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5701.237995] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5703.062458] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5706.485215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5706.527806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5706.652264] Lustre: lustre-OST0000: new disk, initializing [ 5706.657349] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5706.711624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5709.378515] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5709.508718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5709.513720] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5709.559660] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5713.356571] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5715.359366] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5715.464949] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5717.396722] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5717.515518] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5720.037689] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5720.038698] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5720.043073] LustreError: Skipped 1 previous similar message [ 5720.049535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5720.054473] Lustre: Skipped 1 previous similar message [ 5724.535133] LustreError: 292436:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5724.613382] Lustre: server umount lustre-OST0000 complete [ 5730.274055] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5730.275701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5730.279447] Lustre: Skipped 2 previous similar messages [ 5732.354938] LustreError: 292637:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5732.357340] LustreError: 292637:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5732.411854] Lustre: server umount lustre-MDT0000 complete [ 5734.279862] LustreError: 291264:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753446171 with bad export cookie 11383641070342661858 [ 5734.284521] LustreError: 291264:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5734.284826] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5734.354471] LustreError: 292838:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5734.358517] LustreError: 292838:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5734.486647] Lustre: server umount lustre-MDT0001 complete [ 5738.414832] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 08:22:54 (1753446174) [ 5741.498525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5741.693109] LustreError: 293508:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5741.701368] LustreError: 293508:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 5741.727342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5743.496814] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5747.168983] LustreError: 293508:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5747.718414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5747.908324] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5749.759756] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5752.391781] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5754.133224] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5757.572948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5757.733720] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5760.247406] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5762.683216] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5774.305051] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5774.308368] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5774.316630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5774.319584] Lustre: Skipped 1 previous similar message [ 5777.378054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5777.378483] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5777.378607] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5777.380855] Lustre: Skipped 1 previous similar message [ 5778.735102] LustreError: 295734:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5778.736923] LustreError: 295734:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5778.816169] Lustre: server umount lustre-OST0000 complete [ 5782.497539] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5782.498906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5782.502293] Lustre: Skipped 1 previous similar message [ 5782.506276] Lustre: Skipped 1 previous similar message [ 5786.494731] LustreError: 295938:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5786.497410] LustreError: 295938:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5786.533536] Lustre: server umount lustre-MDT0000 complete [ 5787.616039] LustreError: 293508:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5787.622096] LustreError: 293508:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 5788.081361] LustreError: 293489:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753446225 with bad export cookie 11383641070342662747 [ 5788.083213] LustreError: 294224:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 5788.084815] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5788.086251] LustreError: 293489:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5788.235220] Lustre: server umount lustre-MDT0001 complete [ 5793.650565] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 5794.836805] Key type lgssc unregistered [ 5794.988085] LNet: 296665:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5796.005559] LNet: Removed LNI 192.168.203.123@tcp [ 5796.360440] Key type .llcrypt unregistered [ 5796.362053] Key type ._llcrypt unregistered [ 5806.694940] Key type ._llcrypt registered [ 5806.696499] Key type .llcrypt registered [ 5806.740300] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 5813.286182] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 5813.613031] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5813.622897] alg: No test for adler32 (adler32-zlib) [ 5814.494829] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 5814.598721] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 5814.601531] LNet: Accept secure, port 988 [ 5816.191183] Key type lgssc registered [ 5816.619539] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5820.139335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5822.742922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5825.018472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5827.418988] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5829.391523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5831.328670] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5834.202759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5834.229065] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5834.238348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5835.350301] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5835.364324] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5835.404479] Lustre: lustre-MDT0000: new disk, initializing [ 5835.442254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5835.452438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5836.941696] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5841.794163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5841.828236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5841.871967] Lustre: 300446:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5841.902643] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5841.905241] Lustre: Skipped 1 previous similar message [ 5841.944645] Lustre: lustre-MDT0001: new disk, initializing [ 5841.987115] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5842.006323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5842.010439] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5843.609772] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5847.109251] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5848.574247] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5851.202037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5851.231903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5851.358590] Lustre: lustre-OST2710: new disk, initializing [ 5851.361843] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 5851.402958] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 5853.736145] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5856.255755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 5856.259867] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 5856.304212] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 5857.057792] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 5859.688761] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5859.717121] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5859.778066] Lustre: lustre-OST03e8: new disk, initializing [ 5859.781184] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 5859.821370] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 5861.958759] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5864.057716] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 5864.451922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 5864.455613] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 5864.486948] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 5867.989475] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 5868.078808] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5869.585642] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 5869.670613] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5871.123981] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 5871.209259] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5872.652983] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 5872.733940] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5874.658571] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5874.659769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5874.663122] Lustre: Skipped 3 previous similar messages [ 5874.665432] Lustre: Skipped 2 previous similar messages [ 5879.778589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5879.781402] Lustre: Skipped 3 previous similar messages [ 5879.937380] LustreError: 304741:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5879.986202] Lustre: server umount lustre-MDT0000 complete [ 5881.411331] LustreError: 300437:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753446318 with bad export cookie 17683326013062221608 [ 5881.415174] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5881.417449] LustreError: 300437:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5881.486554] LustreError: 304942:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5881.490644] LustreError: 304942:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5881.582087] Lustre: server umount lustre-MDT0001 complete [ 5893.106787] LustreError: 305143:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5893.108824] LustreError: 305143:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5893.127444] Lustre: server umount lustre-OST2710 complete [ 5905.011350] LustreError: 305344:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5905.014155] LustreError: 305344:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5905.086562] Lustre: server umount lustre-OST03e8 complete [ 5912.313503] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 5915.986270] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 5919.805086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5922.717628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5925.456111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5927.992963] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5930.649384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5930.679867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5930.782723] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5930.798419] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5930.840908] Lustre: lustre-MDT0000: new disk, initializing [ 5930.884140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5930.894848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5932.346415] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5937.004262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5937.034694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5937.071849] Lustre: 308121:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5937.188912] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5938.680824] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5942.129541] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5943.500076] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5946.050096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5946.074968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5946.176776] Lustre: lustre-OST0000: new disk, initializing [ 5946.178359] Lustre: Skipped 1 previous similar message [ 5946.180141] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5946.182591] Lustre: Skipped 2 previous similar messages [ 5946.213244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5946.216031] Lustre: Skipped 1 previous similar message [ 5948.315986] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5951.615287] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5953.162432] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5954.051402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5954.054891] Lustre: Skipped 1 previous similar message [ 5954.057077] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5954.102939] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5954.287784] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 5955.768947] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5955.854826] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5959.138783] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5959.139428] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5959.141093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5959.142230] LustreError: Skipped 1 previous similar message [ 5959.146958] Lustre: Skipped 1 previous similar message [ 5962.668771] LustreError: 311035:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5962.671245] LustreError: 311035:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5962.743468] Lustre: server umount lustre-OST0000 complete [ 5967.840810] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5967.844740] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5967.851039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5967.853861] Lustre: Skipped 1 previous similar message [ 5970.457715] Lustre: server umount lustre-MDT0000 complete [ 5971.952188] LustreError: 309855:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753446409 with bad export cookie 17683326013062223288 [ 5971.956110] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5971.956342] LustreError: 309855:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5972.031959] LustreError: 311439:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5972.034382] LustreError: 311439:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 5972.137175] Lustre: server umount lustre-MDT0001 complete [ 5975.410491] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 08:26:52 (1753446412) [ 5975.971804] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 5976.680349] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 08:26:53 (1753446413) [ 5977.851388] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 5980.906991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5982.500621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5983.963229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5985.470521] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5985.889581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5988.514661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5988.644890] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5988.654899] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5988.726535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5990.240861] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5993.444444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5993.490091] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5993.500942] Lustre: 313555:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5995.135454] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5997.146518] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5998.509940] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6001.121632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6001.153731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6001.247158] Lustre: Denying initial registration attempt from nid 192.168.203.123@tcp, specified as failover [ 6001.251104] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 6001.255340] LustreError: 315160:0:(tgt_mount.c:2272:server_fill_super()) Unable to start targets: -99 [ 6001.258358] LustreError: 315160:0:(tgt_mount.c:1766:server_put_super()) no obd lustre-OST0000 [ 6001.261274] LustreError: 315160:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 6001.320348] Lustre: server umount lustre-OST0000 complete [ 6001.322171] LustreError: 315160:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 6004.192431] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6004.195562] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6004.200321] Lustre: Skipped 1 previous similar message [ 6004.202749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6004.205099] Lustre: Skipped 2 previous similar messages [ 6010.049794] LustreError: 315527:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 6010.052242] LustreError: 315527:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6010.082807] Lustre: server umount lustre-MDT0000 complete [ 6011.492377] LustreError: 313546:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753446448 with bad export cookie 17683326013062224170 [ 6011.494102] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6011.498312] LustreError: 313546:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6011.650821] Lustre: server umount lustre-MDT0001 complete [ 6016.919256] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 6018.168155] Key type lgssc unregistered [ 6018.319598] LNet: 316255:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6019.365570] LNet: Removed LNI 192.168.203.123@tcp [ 6019.731667] Key type .llcrypt unregistered [ 6019.732918] Key type ._llcrypt unregistered [ 6023.787306] Key type ._llcrypt registered [ 6023.788958] Key type .llcrypt registered [ 6023.833271] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 6024.301816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6024.364724] alg: No test for adler32 (adler32-zlib) [ 6025.234902] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 6025.336771] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 6025.339302] LNet: Accept secure, port 988 [ 6026.927232] Key type lgssc registered [ 6027.353349] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6030.021058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6031.565369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6033.084416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6034.587820] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6034.987079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6042.235935] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 6045.644873] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6045.653614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6046.769993] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6046.782059] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6046.848084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6048.313493] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6051.498594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6051.553319] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6051.566819] Lustre: 318964:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6051.674824] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6053.076787] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6055.015144] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6056.281764] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6058.914643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6058.949498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6059.038297] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6059.133125] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6061.329678] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6063.495961] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6064.288093] LustreError: 321220:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6064.350913] Lustre: server umount lustre-OST0000 complete [ 6067.167783] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6067.170348] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6067.175215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6069.728867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6069.728876] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6071.869330] LustreError: 321429:0:(obd_class.h:479:obd_check_dev()) Device 17 not setup [ 6071.871862] LustreError: 321429:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6071.903096] Lustre: server umount lustre-MDT0000 complete [ 6073.293393] LustreError: 320596:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753446510 with bad export cookie 16297644507334922039 [ 6073.296238] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6073.299062] LustreError: 320596:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6073.367754] LustreError: 321631:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6073.370346] LustreError: 321631:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6073.457138] Lustre: server umount lustre-MDT0001 complete [ 6077.613603] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 6078.824780] Key type lgssc unregistered [ 6078.972519] LNet: 322157:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6080.037492] LNet: Removed LNI 192.168.203.123@tcp [ 6080.403157] Key type .llcrypt unregistered [ 6080.404898] Key type ._llcrypt unregistered [ 6090.653358] Key type ._llcrypt registered [ 6090.654303] Key type .llcrypt registered [ 6090.697994] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 6091.206398] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6091.215090] alg: No test for adler32 (adler32-zlib) [ 6092.083821] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 6092.180081] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 6092.182593] LNet: Accept secure, port 988 [ 6093.775187] Key type lgssc registered [ 6094.169804] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6097.139595] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6097.148204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6098.385915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6099.926627] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6103.274649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6103.406660] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6104.868319] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6107.016312] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6108.500439] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6111.052509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6111.172937] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6113.129526] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6114.922862] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6128.609543] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6128.611505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6128.614288] Lustre: Skipped 2 previous similar messages [ 6128.616972] Lustre: Skipped 1 previous similar message [ 6130.646793] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.23@tcp (stopping) [ 6130.649812] Lustre: Skipped 1 previous similar message [ 6133.730867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6133.733341] Lustre: Skipped 2 previous similar messages [ 6133.952795] LustreError: 325767:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6133.980260] Lustre: server umount lustre-MDT0000 complete [ 6135.259307] LustreError: 323500:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753446572 with bad export cookie 15645245629580479758 [ 6135.262493] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6135.263563] LustreError: 323500:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6135.756120] LustreError: 323515:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6135.763984] LustreError: 323515:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 6135.768323] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.23@tcp (stopping) [ 6138.847944] LustreError: 323516:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6138.848251] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6140.876397] LustreError: 323516:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6140.876575] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.23@tcp (stopping) [ 6140.884464] Lustre: Skipped 1 previous similar message [ 6141.373488] LustreError: 325970:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6141.375892] LustreError: 325970:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6141.471676] Lustre: server umount lustre-MDT0001 complete [ 6143.853821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6146.959200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6160.351851] LustreError: 322834:0:(client.c:1385:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000001ff05af0 x1838621947616128/t0(0) o250->MGC192.168.203.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6160.471493] LustreError: 325114:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6160.478105] LustreError: 325114:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 6160.497808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6160.520498] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 6161.990632] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6164.577785] LustreError: 326779:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6164.583318] LustreError: 326779:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 6165.202644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6165.353081] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6166.888604] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6169.016648] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6170.469181] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 6172.739415] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 6174.260337] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6174.924092] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6180.836747] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6180.837308] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6180.842694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6180.847443] Lustre: Skipped 2 previous similar messages [ 6182.453357] LustreError: 328263:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6182.456032] LustreError: 328263:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6182.534098] Lustre: server umount lustre-OST0000 complete [ 6185.953047] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6185.955103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6185.963043] Lustre: Skipped 2 previous similar messages [ 6190.208173] LustreError: 328464:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6190.211026] LustreError: 328464:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6190.240638] Lustre: server umount lustre-MDT0000 complete [ 6191.072093] LustreError: 326783:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6191.079108] LustreError: 326783:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 6191.595134] LustreError: 323498:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753446629 with bad export cookie 15645245629580486870 [ 6191.597912] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6191.598918] LustreError: 323498:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6191.750797] Lustre: server umount lustre-MDT0001 complete [ 6196.825269] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 6197.974411] Key type lgssc unregistered [ 6198.120843] LNet: 329192:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6199.141343] LNet: Removed LNI 192.168.203.123@tcp [ 6199.463510] Key type .llcrypt unregistered [ 6199.465378] Key type ._llcrypt unregistered [ 6209.634294] Key type ._llcrypt registered [ 6209.635576] Key type .llcrypt registered [ 6209.677636] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 6210.115230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6210.145978] alg: No test for adler32 (adler32-zlib) [ 6211.015532] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 6211.105325] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 6211.107655] LNet: Accept secure, port 988 [ 6212.703172] Key type lgssc registered [ 6213.116735] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6216.323698] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6216.333793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6217.553301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6219.106485] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6222.495656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6222.649105] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6224.111901] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6226.210259] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6227.587770] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6232.865066] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6232.870962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6232.873259] Lustre: Skipped 1 previous similar message [ 6238.176482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6238.178888] Lustre: Skipped 1 previous similar message [ 6243.295194] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6243.296610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6243.301816] Lustre: Skipped 1 previous similar message [ 6243.332234] LustreError: 331933:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6243.363765] Lustre: server umount lustre-MDT0000 complete [ 6244.885772] LustreError: 331936:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753446682 with bad export cookie 17465585405947223551 [ 6244.890315] LustreError: 331936:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6244.890443] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6244.965217] LustreError: 332138:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6244.967745] LustreError: 332138:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6245.060993] Lustre: server umount lustre-MDT0001 complete [ 6250.573833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6250.690095] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6250.704359] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6250.790316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6252.272322] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6255.529258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6255.578364] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6255.593412] Lustre: 332907:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6255.723483] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6257.184353] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6259.192068] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6260.518315] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6263.255844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6263.334056] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 6263.338419] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 6263.342916] LustreError: 334554:0:(tgt_mount.c:2272:server_fill_super()) Unable to start targets: -2 [ 6263.346391] LustreError: 334554:0:(tgt_mount.c:1766:server_put_super()) no obd lustre-OST0000 [ 6263.349041] LustreError: 334554:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 6263.400632] Lustre: server umount lustre-OST0000 complete [ 6263.402399] LustreError: 334554:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 6266.616046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6266.699613] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6266.768582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6268.860270] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6270.913622] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6273.569657] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 6273.996605] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6274.025100] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6274.054634] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6274.091090] Lustre: lustre-OST0001: new disk, initializing [ 6274.093276] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6276.253663] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6278.594290] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6279.766195] LustreError: 336652:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 6279.768582] LustreError: 336652:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6279.793161] Lustre: server umount lustre-OST0001 complete [ 6283.248194] LustreError: 335022:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6283.249848] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6283.251996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6283.255644] LustreError: 335022:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 6283.260048] Lustre: Skipped 2 previous similar messages [ 6287.213349] LustreError: 336852:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6287.218247] LustreError: 336852:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6287.292675] Lustre: server umount lustre-OST0000 complete [ 6291.424866] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6291.427902] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6291.433625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6291.436487] Lustre: Skipped 1 previous similar message [ 6293.473384] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6302.689049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6302.691213] Lustre: Skipped 5 previous similar messages [ 6303.199134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6303.236469] LustreError: 337061:0:(obd_class.h:479:obd_check_dev()) Device 24 not setup [ 6303.239189] LustreError: 337061:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6303.269076] Lustre: server umount lustre-MDT0000 complete [ 6304.710287] LustreError: 333626:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753446742 with bad export cookie 17465585405947224090 [ 6304.713267] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6304.714661] LustreError: 333626:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6304.898509] Lustre: server umount lustre-MDT0001 complete [ 6309.066168] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 6310.265513] Key type lgssc unregistered [ 6310.417166] LNet: 337789:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6311.461465] LNet: Removed LNI 192.168.203.123@tcp [ 6311.839730] Key type .llcrypt unregistered [ 6311.841154] Key type ._llcrypt unregistered [ 6323.715728] Key type ._llcrypt registered [ 6323.717028] Key type .llcrypt registered [ 6323.757633] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 6330.426226] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 6330.742770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6330.788057] alg: No test for adler32 (adler32-zlib) [ 6331.650754] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 6331.752810] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 6331.755285] LNet: Accept secure, port 988 [ 6333.335163] Key type lgssc registered [ 6333.687936] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6336.819442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6339.203628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6341.469944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6343.925330] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6346.622311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6346.645492] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6346.654797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6347.765945] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6347.786082] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6347.826798] Lustre: lustre-MDT0000: new disk, initializing [ 6347.867794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6347.877528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6349.464504] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6353.980413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6354.014618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6354.055839] Lustre: 341694:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6354.084783] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6354.087210] Lustre: Skipped 1 previous similar message [ 6354.126284] Lustre: lustre-MDT0001: new disk, initializing [ 6354.171624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6354.188459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6354.194127] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6355.728701] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6359.407815] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6360.897457] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6363.499682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6363.531235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6363.644774] Lustre: lustre-OST0000: new disk, initializing [ 6363.647244] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6363.679878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6365.704473] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6369.018997] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6369.278548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6369.282982] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6369.323053] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6370.659299] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6370.747953] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6372.186100] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6372.283359] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6374.370767] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6374.370969] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6374.371916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6374.381855] Lustre: Skipped 1 previous similar message [ 6375.023062] LustreError: 344602:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6375.084759] Lustre: server umount lustre-OST0000 complete [ 6379.488774] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6379.489663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6379.493943] Lustre: Skipped 1 previous similar message [ 6379.496460] Lustre: Skipped 1 previous similar message [ 6382.590125] LustreError: 344803:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6382.592057] LustreError: 344803:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6382.622324] Lustre: server umount lustre-MDT0000 complete [ 6384.096790] LustreError: 341685:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753446821 with bad export cookie 16122565699752553025 [ 6384.101401] LustreError: 341685:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6384.101485] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6384.168232] LustreError: 345005:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6384.170025] LustreError: 345005:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6384.258218] Lustre: server umount lustre-MDT0001 complete [ 6387.694573] Lustre: DEBUG MARKER: == conf-sanity test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 08:33:44 (1753446824) [ 6390.108566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6392.654533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6401.788953] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 6405.250711] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 6408.925041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6411.388256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6413.641293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6415.964093] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6418.434853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6418.457658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6418.551197] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6418.565480] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6418.607591] Lustre: lustre-MDT0000: new disk, initializing [ 6418.636484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6418.645715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6420.109532] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6424.748286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6424.773348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6424.812613] Lustre: 348754:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6424.838755] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6424.841147] Lustre: Skipped 1 previous similar message [ 6424.881271] Lustre: lustre-MDT0001: new disk, initializing [ 6424.919194] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6424.934032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6424.937838] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6426.350679] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6429.802555] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6431.157323] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6433.525172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6433.551959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6433.652220] Lustre: lustre-OST0000: new disk, initializing [ 6433.654932] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6433.685742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6435.706197] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6438.395230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6438.399118] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6438.430063] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6438.872083] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6440.311342] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6440.391330] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6441.801836] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6441.882481] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6443.490564] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6443.492430] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6443.493465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6443.494861] Lustre: Skipped 1 previous similar message [ 6443.501423] LustreError: Skipped 2 previous similar messages [ 6444.523135] LustreError: 351660:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6444.525121] LustreError: 351660:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6444.585739] Lustre: server umount lustre-OST0000 complete [ 6448.608685] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6448.609336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6448.612966] Lustre: Skipped 1 previous similar message [ 6448.615217] Lustre: Skipped 1 previous similar message [ 6453.728134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6453.730186] Lustre: Skipped 1 previous similar message [ 6460.383137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6460.414434] LustreError: 351860:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6460.416633] LustreError: 351860:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6460.451863] Lustre: server umount lustre-MDT0000 complete [ 6461.694862] LustreError: 348747:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753446899 with bad export cookie 16122565699752553893 [ 6461.697514] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6461.698372] LustreError: 348747:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6461.833191] Lustre: server umount lustre-MDT0001 complete [ 6464.628693] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 08:35:01 (1753446901) [ 6467.138482] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 08:35:03 (1753446903) [ 6469.953739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6470.115540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6471.435247] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6474.389326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6475.894162] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6477.736556] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6478.997657] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6481.286524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6483.284658] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6485.155611] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6499.138488] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 6499.733165] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 6500.312349] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 6500.907839] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 6501.574104] Lustre: Failing over lustre-MDT0000 [ 6501.713901] LustreError: 355520:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6501.716013] LustreError: 355520:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6501.750607] Lustre: server umount lustre-MDT0000 complete [ 6504.929818] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6504.933185] LustreError: 355217:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6504.934847] Lustre: Skipped 2 previous similar messages [ 6504.939988] LustreError: 355217:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 6505.940673] LustreError: 353073:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6510.049275] LustreError: 353079:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6510.053356] LustreError: 353079:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 6514.764916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6514.807781] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6514.887567] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6514.891047] Lustre: Skipped 2 previous similar messages [ 6514.923793] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6516.172889] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6516.216423] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6520.297550] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 6520.303920] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 6520.319087] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6521.467604] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6522.018382] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6523.915878] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 6524.565873] Lustre: Failing over lustre-MDT0000 [ 6524.678315] LustreError: 356593:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6524.680342] LustreError: 356593:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6524.709724] Lustre: server umount lustre-MDT0000 complete [ 6525.409302] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6525.409809] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6525.411035] LustreError: 353784:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6525.411042] LustreError: 353784:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 6525.413964] Lustre: Skipped 1 previous similar message [ 6530.529332] LustreError: 353784:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6530.536277] LustreError: 353784:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 5 previous similar messages [ 6537.630415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6537.666252] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6537.770977] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6539.070138] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6541.772713] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6542.822039] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 6542.824277] Lustre: Skipped 2 previous similar messages [ 6542.829376] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 6542.844593] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6543.937118] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6544.447591] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6547.936996] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6547.938964] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6547.940476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6547.942923] Lustre: Skipped 2 previous similar messages [ 6547.944510] Lustre: Skipped 3 previous similar messages [ 6552.790125] Lustre: server umount lustre-MDT0000 complete [ 6553.057734] LustreError: 353079:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6553.063386] LustreError: 353079:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 7 previous similar messages [ 6554.045056] LustreError: 353059:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753446991 with bad export cookie 16122565699752556560 [ 6554.047965] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6554.048946] LustreError: 353059:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6554.193578] Lustre: server umount lustre-MDT0001 complete [ 6565.744204] LustreError: 358018:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6565.746685] LustreError: 358018:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 6565.811758] Lustre: server umount lustre-OST0000 complete [ 6570.815377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6570.961449] LustreError: 358846:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6572.285927] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6575.198542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6576.728982] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6578.560492] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6579.754926] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6582.081137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6582.191202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6582.194431] Lustre: Skipped 3 previous similar messages [ 6584.142779] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6586.025702] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6597.546109] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6605.791892] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6605.794912] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6605.799922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6609.003881] Lustre: server umount lustre-OST0000 complete [ 6616.531286] Lustre: server umount lustre-MDT0000 complete [ 6617.735051] LustreError: 358825:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753447055 with bad export cookie 16122565699752557267 [ 6617.737665] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6617.738821] LustreError: 358825:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6617.893713] Lustre: server umount lustre-MDT0001 complete [ 6622.610589] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 6623.822182] Key type lgssc unregistered [ 6623.961340] LNet: 362025:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6624.997389] LNet: Removed LNI 192.168.203.123@tcp [ 6625.310519] Key type .llcrypt unregistered [ 6625.311873] Key type ._llcrypt unregistered [ 6634.714181] Key type ._llcrypt registered [ 6634.715167] Key type .llcrypt registered [ 6634.749326] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 6635.145620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6635.157945] alg: No test for adler32 (adler32-zlib) [ 6636.016146] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 6636.109220] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 6636.112562] LNet: Accept secure, port 988 [ 6637.695191] Key type lgssc registered [ 6638.059677] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6640.731514] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6640.737556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6641.877925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6643.058783] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6645.991488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6646.131556] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6647.420826] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6649.209922] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6650.369125] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6652.646872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6652.763297] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6654.686092] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6656.454299] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6667.110750] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6671.839976] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6671.843440] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6671.849196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6672.353302] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6672.353620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6672.357861] Lustre: Skipped 1 previous similar message [ 6672.360129] Lustre: Skipped 1 previous similar message [ 6677.246338] LustreError: 365738:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6677.273882] Lustre: server umount lustre-MDT0000 complete [ 6677.472495] LustreError: 363484:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6677.477841] LustreError: 363484:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 6678.552340] LustreError: 363469:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753447116 with bad export cookie 12583404468669467491 [ 6678.555729] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6678.556450] LustreError: 363469:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6678.624180] LustreError: 365940:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6678.626260] LustreError: 365940:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6678.706568] Lustre: server umount lustre-MDT0001 complete [ 6690.292188] LustreError: 366141:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6690.294872] LustreError: 366141:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6690.370211] Lustre: server umount lustre-OST0000 complete [ 6694.929172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6695.085299] LustreError: 366870:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6695.108031] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6696.407307] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6699.234276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6699.354045] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6700.756675] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6702.566737] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6703.844355] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6706.177991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6708.139502] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6709.896083] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6720.614071] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 6735.328299] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6735.330811] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6735.334782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6738.553744] LustreError: 369219:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6738.555432] LustreError: 369219:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6738.584149] Lustre: server umount lustre-MDT0000 complete [ 6739.833150] LustreError: 366851:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753447177 with bad export cookie 12583404468669468590 [ 6739.836159] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6739.836395] LustreError: 366851:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6739.974087] Lustre: server umount lustre-MDT0001 complete [ 6751.472151] LustreError: 369624:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6751.474308] LustreError: 369624:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 6751.532248] Lustre: server umount lustre-OST0000 complete [ 6756.500962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6756.640911] LustreError: 370452:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6756.662942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6756.666142] Lustre: Skipped 1 previous similar message [ 6757.868502] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6760.668445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6762.105823] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6763.868144] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6765.175756] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6767.519696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6769.612110] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6771.508812] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6781.989177] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 6791.135814] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6791.137581] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6791.140279] Lustre: Skipped 2 previous similar messages [ 6791.141627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6791.142996] Lustre: Skipped 3 previous similar messages [ 6794.221750] LustreError: 372703:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6794.223861] LustreError: 372703:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6794.298110] Lustre: server umount lustre-OST0000 complete [ 6796.768089] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6796.770581] LustreError: Skipped 1 previous similar message [ 6796.771995] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6796.775703] Lustre: Skipped 1 previous similar message [ 6796.777559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6796.779069] Lustre: Skipped 2 previous similar messages [ 6801.627570] Lustre: server umount lustre-MDT0000 complete [ 6802.400942] LustreError: 370470:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6802.407886] LustreError: 370470:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 6802.941384] LustreError: 372603:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753447240 with bad export cookie 12583404468669469598 [ 6802.946584] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6802.946896] LustreError: 372603:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6803.099819] Lustre: server umount lustre-MDT0001 complete [ 6808.123977] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 6809.237056] Key type lgssc unregistered [ 6809.369451] LNet: 373631:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6810.405410] LNet: Removed LNI 192.168.203.123@tcp [ 6810.734186] Key type .llcrypt unregistered [ 6810.735430] Key type ._llcrypt unregistered [ 6819.869152] Key type ._llcrypt registered [ 6819.870697] Key type .llcrypt registered [ 6819.905570] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 6820.206367] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6820.252164] alg: No test for adler32 (adler32-zlib) [ 6821.100808] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 6821.183462] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 6821.184768] LNet: Accept secure, port 988 [ 6822.759209] Key type lgssc registered [ 6823.112302] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6825.564554] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6825.568951] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6825.572705] LustreError: 375016:0:(osd_handler.c:8370:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 6825.581290] LustreError: 375016:0:(obd_config.c:779:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 6825.584638] LustreError: 375016:0:(obd_mount.c:193:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 6825.588516] LustreError: 375016:0:(tgt_mount.c:2240:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 6825.592976] LustreError: 375016:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6828.111769] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6828.114523] LustreError: 375414:0:(osd_handler.c:8370:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 6828.123983] LustreError: 375414:0:(obd_config.c:779:class_setup()) setup lustre-OST0000-osd failed (-22) [ 6828.126217] LustreError: 375414:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 6828.129460] LustreError: 375414:0:(tgt_mount.c:2240:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 6828.131501] LustreError: 375414:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6834.605424] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 6835.687122] Key type lgssc unregistered [ 6835.831493] LNet: 376393:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6835.840453] LNet: Removed LNI 192.168.203.123@tcp [ 6836.140492] Key type .llcrypt unregistered [ 6836.141917] Key type ._llcrypt unregistered [ 6842.227974] Key type ._llcrypt registered [ 6842.229540] Key type .llcrypt registered [ 6842.268812] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 6848.620946] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 6849.035482] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6849.053518] alg: No test for adler32 (adler32-zlib) [ 6849.912501] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 6850.009697] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 6850.012444] LNet: Accept secure, port 988 [ 6851.591113] Key type lgssc registered [ 6851.961338] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6855.117096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6857.393466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6859.406430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6861.548311] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6863.938865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6863.957478] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6863.964758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6865.053851] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6865.067488] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6865.107765] Lustre: lustre-MDT0000: new disk, initializing [ 6865.137119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6865.147225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6866.437030] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6870.468466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6870.491670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6870.524855] Lustre: 379481:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6870.549506] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6870.551749] Lustre: Skipped 1 previous similar message [ 6870.584411] Lustre: lustre-MDT0001: new disk, initializing [ 6870.614236] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6870.625488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6870.627881] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6871.845980] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6874.773625] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6875.966095] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6878.129478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6878.160071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6878.273562] Lustre: lustre-OST0000: new disk, initializing [ 6878.276272] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6878.313501] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6880.140143] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6883.215919] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6884.623650] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6886.389761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6886.392711] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6886.420503] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6886.745523] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 6888.157982] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6888.246894] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6891.491675] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6891.492431] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6891.497245] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6891.499061] LustreError: Skipped 1 previous similar message [ 6895.019083] LustreError: 382404:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6895.083804] Lustre: server umount lustre-OST0000 complete [ 6896.608034] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6896.611882] Lustre: Skipped 1 previous similar message [ 6896.613605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6896.615710] Lustre: Skipped 1 previous similar message [ 6901.215856] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6901.219130] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6901.223557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6902.521557] LustreError: 382605:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6902.523443] LustreError: 382605:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6902.549152] Lustre: server umount lustre-MDT0000 complete [ 6903.806211] LustreError: 379472:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753447341 with bad export cookie 12617486323673741980 [ 6903.808610] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6903.811122] LustreError: 379472:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6903.874267] LustreError: 382808:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6903.877148] LustreError: 382808:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6903.953488] Lustre: server umount lustre-MDT0001 complete [ 6906.742898] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 6907.315133] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 08:42:24 (1753447344) [ 6909.602506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6909.764047] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6911.049407] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6913.821287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6913.940171] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6915.249390] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6917.052713] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6918.237346] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6920.505750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6922.572605] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6924.290414] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6926.511212] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6926.534631] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6926.590178] Lustre: lustre-OST0001: new disk, initializing [ 6926.592098] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6926.623230] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6926.625916] Lustre: Skipped 1 previous similar message [ 6928.487178] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 6931.523353] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6934.013532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6934.016299] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6934.040604] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6939.105097] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6939.106242] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6939.107884] LustreError: Skipped 1 previous similar message [ 6939.113581] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6939.117251] Lustre: Skipped 2 previous similar messages [ 6943.788883] LustreError: 386700:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 6943.790918] LustreError: 386700:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6943.808572] Lustre: server umount lustre-OST0001 complete [ 6945.254345] LustreError: 385112:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6945.258149] LustreError: 385112:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 6950.369715] LustreError: 385112:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6950.374613] LustreError: 385112:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 6955.492925] LustreError: 385612:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6955.500622] LustreError: 385612:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 6959.583863] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6959.585808] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6959.589370] Lustre: Skipped 1 previous similar message [ 6959.591979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6959.594558] Lustre: Skipped 3 previous similar messages [ 6960.611333] LustreError: 385111:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6960.617791] LustreError: 385111:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 6965.730798] LustreError: 385612:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6965.734957] LustreError: 385612:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 6970.335146] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6970.350244] LustreError: 386922:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6970.352819] LustreError: 386922:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6970.422380] Lustre: server umount lustre-OST0000 complete [ 6975.455845] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6975.458040] LustreError: Skipped 1 previous similar message [ 6975.459327] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6975.461952] Lustre: Skipped 1 previous similar message [ 6975.463510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6975.465097] Lustre: Skipped 4 previous similar messages [ 6977.810491] Lustre: server umount lustre-MDT0000 complete [ 6978.971184] LustreError: 383507:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753447416 with bad export cookie 12617486323673742848 [ 6978.973369] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6978.976041] LustreError: 383507:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6993.375132] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6993.410193] LustreError: 387325:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 6993.411978] LustreError: 387325:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 6993.492911] Lustre: server umount lustre-MDT0001 complete [ 6998.111221] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 6999.278382] Key type lgssc unregistered [ 6999.425464] LNet: 387853:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7000.485466] LNet: Removed LNI 192.168.203.123@tcp [ 7000.830443] Key type .llcrypt unregistered [ 7000.832073] Key type ._llcrypt unregistered [ 7012.322584] Key type ._llcrypt registered [ 7012.323887] Key type .llcrypt registered [ 7012.365826] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 7018.565680] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 7019.027823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7019.036885] alg: No test for adler32 (adler32-zlib) [ 7019.906678] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 7020.001872] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 7020.004505] LNet: Accept secure, port 988 [ 7021.599175] Key type lgssc registered [ 7021.990186] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7025.174653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7027.625740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7029.758804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7031.965080] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7034.235339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7034.255486] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7034.261389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7035.346506] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7035.358376] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7035.394406] Lustre: lustre-MDT0000: new disk, initializing [ 7035.418334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7035.425530] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7036.756880] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7040.934678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7040.957888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7040.993552] Lustre: 391756:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7041.018076] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7041.020295] Lustre: Skipped 1 previous similar message [ 7041.058527] Lustre: lustre-MDT0001: new disk, initializing [ 7041.092922] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7041.106534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7041.110613] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7042.476690] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7045.584595] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7046.797372] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7049.035592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7049.065267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7049.166582] Lustre: lustre-OST0000: new disk, initializing [ 7049.169495] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7049.201640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7049.781451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7049.785613] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7049.818856] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7051.157959] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7054.270901] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7055.654691] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7055.737428] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7057.070411] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7057.162366] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7059.940018] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7059.940690] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7059.942299] LustreError: Skipped 1 previous similar message [ 7059.947992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7059.950319] Lustre: Skipped 1 previous similar message [ 7063.852062] LustreError: 394663:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7063.920929] Lustre: server umount lustre-OST0000 complete [ 7066.592188] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7066.594974] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7066.599191] Lustre: Skipped 1 previous similar message [ 7066.601242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7070.176109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7070.176593] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7071.289788] LustreError: 394865:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7071.291706] LustreError: 394865:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7071.315334] Lustre: server umount lustre-MDT0000 complete [ 7072.528111] LustreError: 391748:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753447510 with bad export cookie 9076482645237787091 [ 7072.531150] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7072.531934] LustreError: 391748:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7072.597269] LustreError: 395066:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7072.600106] LustreError: 395066:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7072.675576] Lustre: server umount lustre-MDT0001 complete [ 7075.390657] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 08:45:12 (1753447512) [ 7078.423523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7081.114670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7081.244142] Lustre: lustre-MDT0000: new disk, initializing [ 7081.273884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7082.476928] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7085.169917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7085.275188] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7086.489321] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7088.208407] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7089.383526] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7090.061140] LustreError: 396360:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7090.064227] LustreError: 396360:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7090.067526] LustreError: 396360:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 7090.656340] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7090.656422] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7090.656876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7090.656880] Lustre: Skipped 1 previous similar message [ 7090.663915] Lustre: Skipped 1 previous similar message [ 7094.752082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7094.753860] Lustre: Skipped 2 previous similar messages [ 7096.187064] LustreError: 397662:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7096.189936] LustreError: 397662:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7096.214972] Lustre: server umount lustre-MDT0000 complete [ 7097.474146] LustreError: 397044:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753447534 with bad export cookie 9076482645237787959 [ 7097.477074] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7097.478264] LustreError: 397044:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7097.608833] Lustre: server umount lustre-MDT0001 complete [ 7100.640171] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 08:45:37 (1753447537) [ 7102.845191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7102.993206] LustreError: 398584:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7102.999196] LustreError: 398584:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 7103.017811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7104.291619] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7107.109271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7108.522692] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7110.325849] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7111.496389] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7113.626675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7113.696794] LustreError: 400168:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7113.703985] LustreError: 400168:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 7115.630496] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7117.396601] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7121.019499] Lustre: Permanently deactivating lustre-OST0000 [ 7121.024277] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7126.512505] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7128.080399] LustreError: 401254:0:(mgs_llog.c:1937:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 7128.083237] LustreError: 401254:0:(mgs_llog.c:1991:mgs_replace_nids()) Only MGS is allowed to be started [ 7128.085363] LustreError: 401254:0:(mgs_handler.c:1032:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7128.673605] LustreError: 401302:0:(mgs_llog.c:1991:mgs_replace_nids()) Only MGS is allowed to be started [ 7128.676057] LustreError: 401302:0:(mgs_handler.c:1032:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7135.404305] LustreError: 401399:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7135.406570] LustreError: 401399:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 7135.471360] Lustre: server umount lustre-OST0000 complete [ 7136.373622] LustreError: 401553:0:(mgs_llog.c:1991:mgs_replace_nids()) Only MGS is allowed to be started [ 7136.377345] LustreError: 401553:0:(mgs_handler.c:1032:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7137.248634] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7137.249157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7137.251958] Lustre: Skipped 1 previous similar message [ 7147.491247] LustreError: 401651:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7147.493744] LustreError: 401651:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7147.516879] Lustre: server umount lustre-MDT0000 complete [ 7148.712045] LustreError: 399294:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753447586 with bad export cookie 9076482645237788477 [ 7148.714661] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7148.715797] LustreError: 399294:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7159.345322] Lustre: server umount lustre-MDT0001 complete [ 7162.239569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7163.516513] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7164.452887] LustreError: 402689:0:(mgs_handler.c:1032:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 7166.571720] Lustre: 403028:0:(mgs_llog.c:1698:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.123@tcp [ 7167.231534] LustreError: 403125:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7167.233457] LustreError: 403125:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 7167.255691] Lustre: server umount MGS complete [ 7171.023559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7171.165084] LustreError: 403752:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7171.169947] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7171.173056] Lustre: Skipped 1 previous similar message [ 7171.192103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7171.194140] Lustre: Skipped 2 previous similar messages [ 7172.400544] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7175.084353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7175.168245] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7176.437898] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7178.199557] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7179.412839] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7180.047304] Lustre: Permanently reactivating lustre-OST0000 [ 7180.052552] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7180.054502] Lustre: Skipped 2 previous similar messages [ 7201.759122] LustreError: 405159:0:(osp_dev.c:694:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 7204.125875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7204.235912] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7204.238453] Lustre: Skipped 1 previous similar message [ 7205.990347] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7207.753900] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7221.729682] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7221.730877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7221.731133] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7221.733135] Lustre: Skipped 1 previous similar message [ 7221.735097] Lustre: Skipped 5 previous similar messages [ 7225.387499] LustreError: 406973:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7225.389695] LustreError: 406973:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 7225.454948] Lustre: server umount lustre-OST0000 complete [ 7231.455942] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7231.459790] LustreError: Skipped 1 previous similar message [ 7232.854316] Lustre: server umount lustre-MDT0000 complete [ 7234.181022] LustreError: 403732:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753447671 with bad export cookie 9076482645237789422 [ 7234.183613] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7234.184687] LustreError: 403732:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7234.322080] Lustre: server umount lustre-MDT0001 complete [ 7238.904780] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 7239.940419] Key type lgssc unregistered [ 7240.070495] LNet: 407907:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7241.125401] LNet: Removed LNI 192.168.203.123@tcp [ 7241.429614] Key type .llcrypt unregistered [ 7241.430608] Key type ._llcrypt unregistered [ 7247.293682] Key type ._llcrypt registered [ 7247.294606] Key type .llcrypt registered [ 7247.324714] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 7253.568901] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 7253.915811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7253.955524] alg: No test for adler32 (adler32-zlib) [ 7254.822122] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 7254.908435] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 7254.910488] LNet: Accept secure, port 988 [ 7256.495151] Key type lgssc registered [ 7256.868941] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7259.995939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7262.310845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7264.348980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7266.465456] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7268.772159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7268.791128] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7268.801652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7269.900944] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7269.914756] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7269.951512] Lustre: lustre-MDT0000: new disk, initializing [ 7269.978738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7269.987242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7271.290395] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7275.622974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7275.652475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7275.692481] Lustre: 410993:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7275.721364] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7275.723587] Lustre: Skipped 1 previous similar message [ 7275.759249] Lustre: lustre-MDT0001: new disk, initializing [ 7275.799862] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7275.814341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7275.819340] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7277.244672] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7280.378415] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7281.642418] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7283.999970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7284.030309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7284.136056] Lustre: lustre-OST0000: new disk, initializing [ 7284.138596] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7284.173025] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7284.434863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7284.438887] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7284.483814] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7286.101672] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7289.196628] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7290.651610] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7290.740652] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7292.115315] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7292.207852] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7294.948924] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7294.949062] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7294.952196] LustreError: Skipped 1 previous similar message [ 7294.958435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7294.961451] Lustre: Skipped 1 previous similar message [ 7294.969117] LustreError: 413902:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7295.029680] Lustre: server umount lustre-OST0000 complete [ 7296.480143] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7296.482198] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7296.485233] Lustre: Skipped 1 previous similar message [ 7296.487228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7300.064940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7300.065700] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7302.459380] LustreError: 414103:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7302.461593] LustreError: 414103:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7302.491364] Lustre: server umount lustre-MDT0000 complete [ 7303.681434] LustreError: 412728:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753447741 with bad export cookie 6767562155496405091 [ 7303.683652] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7303.687072] LustreError: 412728:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7303.753723] LustreError: 414304:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7303.755573] LustreError: 414304:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7303.852839] Lustre: server umount lustre-MDT0001 complete [ 7306.483595] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 7307.048158] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 08:49:03 (1753447743) [ 7309.402065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7309.592864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7310.906638] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7313.731438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7313.854774] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7315.142401] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7316.966123] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7318.123052] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7320.343734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7322.258493] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7323.981795] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7324.643607] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 7335.448813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 7338.975907] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7338.979072] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7338.984599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7338.987060] Lustre: Skipped 1 previous similar message [ 7343.017237] LustreError: 417461:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7343.019069] LustreError: 417461:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7343.077489] Lustre: server umount lustre-OST0000 complete [ 7344.607680] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7344.610080] LustreError: Skipped 1 previous similar message [ 7344.611409] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7344.614947] Lustre: Skipped 1 previous similar message [ 7344.616558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7344.618133] Lustre: Skipped 2 previous similar messages [ 7350.333349] LustreError: 417664:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7350.335295] LustreError: 417664:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7350.367076] Lustre: server umount lustre-MDT0000 complete [ 7351.626436] LustreError: 417665:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753447789 with bad export cookie 6767562155496405959 [ 7351.631397] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7351.795362] Lustre: server umount lustre-MDT0001 complete [ 7356.432449] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 7357.545911] Key type lgssc unregistered [ 7357.696352] LNet: 418392:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7358.757377] LNet: Removed LNI 192.168.203.123@tcp [ 7359.104662] Key type .llcrypt unregistered [ 7359.106261] Key type ._llcrypt unregistered [ 7366.468668] Key type ._llcrypt registered [ 7366.469510] Key type .llcrypt registered [ 7366.509486] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 7366.611488] Key type .llcrypt unregistered [ 7366.613081] Key type ._llcrypt unregistered [ 7373.326462] Key type ._llcrypt registered [ 7373.328149] Key type .llcrypt registered [ 7373.368695] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 7373.850299] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7373.869289] alg: No test for adler32 (adler32-zlib) [ 7374.728624] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 7374.820593] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 7374.822812] LNet: Accept secure, port 988 [ 7376.407120] Key type lgssc registered [ 7376.787616] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7379.460482] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7379.466972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7380.620554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7381.956860] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7384.756960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7384.868892] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7386.806462] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7388.570283] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7390.177341] LustreError: 420543:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7390.182220] LustreError: 420543:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 7390.815173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7390.927124] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7391.982102] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7392.166956] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7396.320572] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7396.322836] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7396.326865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7397.345645] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7397.347872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7397.348378] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7397.355489] Lustre: Skipped 1 previous similar message [ 7401.005595] LustreError: 422559:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 7401.064436] Lustre: server umount lustre-OST0000 complete [ 7402.464277] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7402.464912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7402.467596] Lustre: Skipped 1 previous similar message [ 7402.470361] Lustre: Skipped 1 previous similar message [ 7407.584722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7407.587434] Lustre: Skipped 1 previous similar message [ 7408.378756] LustreError: 422761:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7408.380595] LustreError: 422761:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7408.404097] Lustre: server umount lustre-MDT0000 complete [ 7409.596154] LustreError: 420503:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753447847 with bad export cookie 7861424428914474067 [ 7409.599584] LustreError: 420503:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7409.599691] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7409.664836] LustreError: 422963:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7409.666577] LustreError: 422963:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7409.746511] Lustre: server umount lustre-MDT0001 complete [ 7414.558805] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 7415.677118] Key type lgssc unregistered [ 7415.812374] LNet: 423489:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7416.869347] LNet: Removed LNI 192.168.203.123@tcp [ 7417.194046] Key type .llcrypt unregistered [ 7417.195449] Key type ._llcrypt unregistered [ 7426.402801] Key type ._llcrypt registered [ 7426.404198] Key type .llcrypt registered [ 7426.452965] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 7426.843638] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7426.851296] alg: No test for adler32 (adler32-zlib) [ 7427.709520] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 7427.801467] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 7427.803043] LNet: Accept secure, port 988 [ 7429.383220] Key type lgssc registered [ 7429.779148] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7432.437594] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7432.443236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7449.951338] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7455.071188] LustreError: 424826:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.123@tcp: failed processing log, type 4: rc = -110 [ 7485.791151] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7487.740468] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7489.589559] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7491.926340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7492.096997] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7492.125266] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 7493.379392] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7496.337171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7496.469685] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7497.839240] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7499.716126] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7501.022646] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7504.867445] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7504.868154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7504.873112] Lustre: Skipped 1 previous similar message [ 7504.875817] Lustre: Skipped 1 previous similar message [ 7509.488096] LustreError: 427091:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 7509.560377] Lustre: server umount lustre-OST0000 complete [ 7512.031940] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7512.035386] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7512.040855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7515.104350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7515.104875] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7516.924906] LustreError: 427293:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 7516.926770] LustreError: 427293:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7516.955544] Lustre: server umount lustre-MDT0000 complete [ 7518.235230] LustreError: 424833:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753447955 with bad export cookie 11585054118057446715 [ 7518.239631] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7518.240946] LustreError: 424833:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7518.305975] LustreError: 427494:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7518.308932] LustreError: 427494:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7518.388272] Lustre: server umount lustre-MDT0001 complete [ 7523.026303] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 7524.180462] Key type lgssc unregistered [ 7524.331479] LNet: 428021:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7525.349316] LNet: Removed LNI 192.168.203.123@tcp [ 7525.670630] Key type .llcrypt unregistered [ 7525.671598] Key type ._llcrypt unregistered [ 7534.456339] Key type ._llcrypt registered [ 7534.457469] Key type .llcrypt registered [ 7534.493191] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 7534.927991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7534.960880] alg: No test for adler32 (adler32-zlib) [ 7535.818594] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 7535.909315] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 7535.912356] LNet: Accept secure, port 988 [ 7537.503143] Key type lgssc registered [ 7537.894791] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7540.430331] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7540.436186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7541.565946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7542.744596] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7545.406563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7545.510066] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7546.777982] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7548.636647] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7549.860638] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7552.219630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7552.337752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7554.276666] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7556.109208] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7566.694119] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 7571.424048] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7571.426335] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7571.431270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7571.938725] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7571.944593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7571.946833] Lustre: Skipped 2 previous similar messages [ 7573.754805] LustreError: 431633:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7573.780187] Lustre: server umount lustre-MDT0000 complete [ 7574.524186] LustreError: 430992:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.23@tcp arrived at 1753448012 with bad export cookie 13718999020244443217 [ 7574.529358] LustreError: 430992:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7575.196969] LustreError: 430992:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753448012 with bad export cookie 13718999020244442566 [ 7575.200321] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7575.202299] LustreError: 430992:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7576.031734] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7576.033545] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7576.036313] Lustre: Skipped 1 previous similar message [ 7576.037741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7577.055956] LustreError: 430093:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7577.061930] LustreError: 430093:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 7580.063901] LustreError: 430093:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7580.064212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7580.070461] LustreError: 430093:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 7580.072318] Lustre: Skipped 1 previous similar message [ 7581.292100] LustreError: 431837:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7581.294365] LustreError: 431837:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7581.367201] Lustre: server umount lustre-OST0000 complete [ 7583.526910] LustreError: 432190:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7583.528723] LustreError: 432190:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7583.602476] Lustre: server umount lustre-MDT0001 complete [ 7587.988716] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 7589.052964] Key type lgssc unregistered [ 7589.186509] LNet: 432716:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7590.245262] LNet: Removed LNI 192.168.203.123@tcp [ 7590.538688] Key type .llcrypt unregistered [ 7590.539551] Key type ._llcrypt unregistered [ 7599.155279] Key type ._llcrypt registered [ 7599.156549] Key type .llcrypt registered [ 7599.198255] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 7599.579610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7599.607132] alg: No test for adler32 (adler32-zlib) [ 7600.455681] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 7600.527947] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 7600.529915] LNet: Accept secure, port 988 [ 7602.103129] Key type lgssc registered [ 7602.406719] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7604.852664] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7604.859935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7606.023061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7607.204462] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7609.832846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7609.961767] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7611.093634] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7612.773300] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7613.926254] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7616.273312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7616.385362] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7617.447296] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 7618.337245] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7620.166884] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7625.697937] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7625.698324] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 7625.702216] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7625.707413] Lustre: Skipped 1 previous similar message [ 7627.731608] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.23@tcp (stopping) [ 7629.185132] LustreError: 436301:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7629.218979] Lustre: server umount lustre-MDT0001 complete [ 7630.816478] LustreError: 434080:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7630.822601] LustreError: 434080:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 7632.864041] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7632.866854] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7632.871229] Lustre: Skipped 1 previous similar message [ 7632.873574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7635.936248] LustreError: 434080:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7635.936821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7637.035083] LustreError: 436507:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7637.038236] LustreError: 436507:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7637.101188] Lustre: server umount lustre-OST0000 complete [ 7638.366740] LustreError: 436708:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7638.368463] LustreError: 436708:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7638.458744] Lustre: server umount lustre-MDT0000 complete [ 7644.048466] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 7645.361349] Key type lgssc unregistered [ 7645.493547] LNet: 437386:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7646.565503] LNet: Removed LNI 192.168.203.123@tcp [ 7646.911075] Key type .llcrypt unregistered [ 7646.912278] Key type ._llcrypt unregistered [ 7654.831842] Key type ._llcrypt registered [ 7654.832820] Key type .llcrypt registered [ 7654.869771] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 7661.172499] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 7661.503801] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7661.541845] alg: No test for adler32 (adler32-zlib) [ 7662.396166] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 7662.473756] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 7662.475299] LNet: Accept secure, port 988 [ 7664.055150] Key type lgssc registered [ 7664.428201] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7667.468078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7669.704749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7671.762693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7673.954730] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7677.186055] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 7680.162591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7682.384530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7682.404450] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7682.409061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7683.497475] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7683.509173] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7683.551649] Lustre: lustre-MDT0000: new disk, initializing [ 7683.578679] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7683.585129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7684.914799] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7688.468964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7690.752964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7690.770566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7690.874716] Lustre: lustre-OST0000: new disk, initializing [ 7690.876909] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7690.879046] Lustre: Skipped 1 previous similar message [ 7690.907246] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7692.795594] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7695.939763] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7698.427525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 7698.430810] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 7698.440483] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 7704.818519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7707.212986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7707.234597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7707.266774] Lustre: 441302:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7707.296945] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7707.332528] Lustre: lustre-MDT0001: new disk, initializing [ 7707.377612] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7707.390122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 7707.393341] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 7708.658805] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7712.267702] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7712.354015] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7713.692920] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7713.789806] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7718.369961] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 7718.371038] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7718.373180] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7718.377102] Lustre: Skipped 1 previous similar message [ 7718.382013] Lustre: Skipped 1 previous similar message [ 7719.891441] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.23@tcp (stopping) [ 7721.468932] LustreError: 444691:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7721.494327] Lustre: server umount lustre-MDT0001 complete [ 7728.610106] LustreError: 442892:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7728.615066] LustreError: 442892:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 7733.731066] LustreError: 442892:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7733.736293] LustreError: 442892:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 7734.239846] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7734.242205] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7734.246421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7738.848809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7738.849177] LustreError: 441313:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7740.012153] LustreError: 444966:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 7740.014355] LustreError: 444966:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7740.093942] Lustre: server umount lustre-OST0000 complete [ 7747.582106] LustreError: 445167:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 7747.585019] LustreError: 445167:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7747.677882] Lustre: server umount lustre-MDT0000 complete [ 7753.614647] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 7754.708190] Key type lgssc unregistered [ 7754.842357] LNet: 445847:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7755.877361] LNet: Removed LNI 192.168.203.123@tcp [ 7756.208552] Key type .llcrypt unregistered [ 7756.209712] Key type ._llcrypt unregistered [ 7769.132683] Key type ._llcrypt registered [ 7769.133681] Key type .llcrypt registered [ 7769.171119] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 7769.281172] Key type .llcrypt unregistered [ 7769.282140] Key type ._llcrypt unregistered [ 7775.932437] Key type ._llcrypt registered [ 7775.934245] Key type .llcrypt registered [ 7775.970664] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 7776.378331] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7776.393910] alg: No test for adler32 (adler32-zlib) [ 7777.255412] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 7777.350599] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 7777.353557] LNet: Accept secure, port 988 [ 7778.935117] Key type lgssc registered [ 7779.301035] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7782.063942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7784.340616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7786.307743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7788.512992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7788.533086] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7788.540309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7789.645418] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7789.659140] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7789.704256] Lustre: lustre-MDT0000: new disk, initializing [ 7789.729480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7789.736532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7791.100297] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7795.419037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7795.439528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7795.479162] Lustre: 449148:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7795.504537] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7795.507185] Lustre: Skipped 1 previous similar message [ 7795.550967] Lustre: lustre-MDT0001: new disk, initializing [ 7795.588563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7795.601762] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7795.605635] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7796.951911] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7800.112189] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7801.453785] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7803.724678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7803.747525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7803.853087] Lustre: lustre-OST0000: new disk, initializing [ 7803.854994] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7803.882895] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7805.755422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7805.760449] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7805.792629] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7805.899313] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7809.071247] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7816.163262] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7816.164858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7816.165391] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7816.166902] Lustre: Skipped 2 previous similar messages [ 7816.168417] Lustre: Skipped 1 previous similar message [ 7817.596711] LustreError: 451571:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7817.622201] Lustre: server umount lustre-MDT0000 complete [ 7818.887344] LustreError: 451485:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753448256 with bad export cookie 3224634026952509247 [ 7818.891770] LustreError: 451485:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7818.891839] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7818.962872] LustreError: 451773:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7818.965806] LustreError: 451773:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7819.062990] Lustre: server umount lustre-MDT0001 complete [ 7829.617081] LustreError: 451973:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7829.620071] LustreError: 451973:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7829.693834] Lustre: server umount lustre-OST0000 complete [ 7833.276257] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 08:57:49 (1753448269) [ 7833.621737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7836.030028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7836.209889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7837.576899] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7840.471263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7840.591254] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7841.942325] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7843.787954] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7845.032592] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7847.394548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7847.427434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7847.494325] Lustre: Found index 0 for lustre-OST0000, updating log [ 7849.510107] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7851.322792] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7860.905462] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 7866.338508] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7866.338598] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7866.341479] LustreError: Skipped 1 previous similar message [ 7866.346756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7866.350274] Lustre: Skipped 1 previous similar message [ 7867.361834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7867.363575] Lustre: Skipped 1 previous similar message [ 7868.972575] LustreError: 455058:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7868.976334] LustreError: 455058:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7869.049985] Lustre: server umount lustre-OST0000 complete [ 7869.992614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7871.456028] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7871.458940] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7871.462892] Lustre: Skipped 1 previous similar message [ 7871.465629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7876.797967] LustreError: 455314:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7876.801627] LustreError: 455314:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7876.829589] Lustre: server umount lustre-MDT0000 complete [ 7878.261956] LustreError: 452780:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753448315 with bad export cookie 3224634026952510626 [ 7878.266091] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7878.269424] LustreError: 452780:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7878.429795] Lustre: server umount lustre-MDT0001 complete [ 7881.380547] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 08:58:38 (1753448318) [ 7881.723698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7882.102837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7882.484053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7882.864741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7883.214317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7883.548136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7883.888520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7884.244571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7884.577192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7884.903991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7885.275794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7885.639762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7885.982481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7886.324696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7886.698142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7887.028632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7887.376640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7887.732114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7888.070504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7888.410670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7888.768628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7889.132313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7889.469396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7889.816615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7890.169445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7890.504522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7890.868754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7891.224717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7891.586582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7891.939348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7892.281177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7892.630446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7892.972523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7893.324431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7893.661176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7894.001710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7894.350398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7894.697980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7895.042242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7895.391872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7895.742531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7896.087491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7896.430490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7896.776576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7897.121665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7897.491356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7897.870174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7898.268349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7898.607254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7898.951243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7899.305500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7902.019954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7902.190242] LustreError: 458896:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7902.195429] LustreError: 458896:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 7902.219686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7902.221423] Lustre: Skipped 1 previous similar message [ 7903.606870] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7906.956219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7908.725791] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7910.739156] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7911.926264] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7914.658280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7914.696822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7914.793079] Lustre: Found index 0 for lustre-OST0000, updating log [ 7916.980035] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7919.061504] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7923.170183] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7923.170383] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7923.176195] Lustre: Skipped 1 previous similar message [ 7923.178197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7923.180474] Lustre: Skipped 4 previous similar messages [ 7925.932072] LustreError: 461186:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7925.934217] LustreError: 461186:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 7926.011576] Lustre: server umount lustre-OST0000 complete [ 7927.775892] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7927.778093] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7931.873149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7931.874943] Lustre: Skipped 8 previous similar messages [ 7933.395612] Lustre: server umount lustre-MDT0000 complete [ 7933.921642] LustreError: 458896:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7933.929027] LustreError: 458896:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 7934.651202] LustreError: 458876:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753448372 with bad export cookie 3224634026952511697 [ 7934.654357] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7934.657312] LustreError: 458876:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7934.791278] Lustre: server umount lustre-MDT0001 complete [ 7935.766132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7941.037936] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 7944.408867] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 7948.164333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7950.453050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7952.484296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7954.585437] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7956.775784] Lustre: DEBUG MARKER: == conf-sanity test 73d: erase + new parameter doesn't corrupt mountdata ========================================================== 08:59:53 (1753448393) [ 7957.141114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7957.469191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7957.808911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7962.940922] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 7965.960271] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 7969.142368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7971.292518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7973.273778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7975.334654] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7977.601781] Lustre: DEBUG MARKER: == conf-sanity test 74: Test per-device adaptive timeout parameters ========================================================== 09:00:14 (1753448414) [ 7979.881653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7979.906847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7980.000174] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7980.014802] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7980.055234] Lustre: lustre-MDT0000: new disk, initializing [ 7980.082341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7980.084313] Lustre: Skipped 2 previous similar messages [ 7980.090101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7981.394722] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7985.580773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7985.605629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7985.636160] Lustre: 467274:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7985.659032] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7985.660746] Lustre: Skipped 1 previous similar message [ 7985.694974] Lustre: lustre-MDT0001: new disk, initializing [ 7985.736646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7985.739509] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7987.018914] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7990.146150] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7991.336293] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7993.662078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7993.687423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7993.777364] Lustre: lustre-OST0000: new disk, initializing [ 7993.780273] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7995.775813] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 7998.938040] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8000.503287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8000.506606] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8000.532085] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8008.107666] loop0: detected capacity change from 0 to 204800000 [ 8008.168372] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8008.172123] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8008.177926] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8008.182077] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8008.205556] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8008.230746] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8008.248410] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8008.277608] blk_update_request: operation not supported error, dev loop0, sector 176912 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8008.309547] blk_update_request: operation not supported error, dev loop0, sector 185112 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8008.337292] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8010.458470] loop0: detected capacity change from 0 to 204800000 [ 8010.524782] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8010.584831] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8012.316498] loop0: detected capacity change from 0 to 204800000 [ 8014.357545] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8014.383823] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8014.426301] Lustre: fs15246-OST0000: new disk, initializing [ 8014.428625] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 8014.445384] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8014.447633] Lustre: Skipped 2 previous similar messages [ 8016.432902] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8020.423699] loop1: detected capacity change from 0 to 204800000 [ 8022.380491] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8022.406790] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8022.438894] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 8022.555372] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8022.596689] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 8023.648873] Lustre: 471815:0:(client.c:1611:after_reply()) @@@ resending request on EINPROGRESS req@000000002be2cfbe x1838623714572544/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1753448477 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 8024.003527] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8031.721736] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 8031.723586] Lustre: Skipped 2 previous similar messages [ 8032.735256] LustreError: 471815:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 8032.738493] LustreError: 471815:0:(fid_request.c:255:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [ 8032.740903] LustreError: 471815:0:(osp_precreate.c:461:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [ 8037.626857] LustreError: 472940:0:(obd_class.h:479:obd_check_dev()) Device 30 not setup [ 8037.628864] LustreError: 472940:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 8037.653542] Lustre: server umount fs15246-MDT0000 complete [ 8040.773374] Lustre: server umount fs15246-OST0000 complete [ 8046.560814] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8046.561035] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8046.563279] LustreError: Skipped 1 previous similar message [ 8046.567938] Lustre: Skipped 1 previous similar message [ 8050.424097] Lustre: server umount lustre-OST0000 complete [ 8057.815733] Lustre: server umount lustre-MDT0000 complete [ 8059.126547] LustreError: 468047:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753448496 with bad export cookie 3224634026952512488 [ 8059.129950] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8059.130240] LustreError: 468047:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8059.287558] Lustre: server umount lustre-MDT0001 complete [ 8064.116852] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 8065.171322] Key type lgssc unregistered [ 8065.315414] LNet: 474879:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8176.933297] LNet: Removed LNI 192.168.203.123@tcp [ 8177.267334] Key type .llcrypt unregistered [ 8177.268663] Key type ._llcrypt unregistered [ 8184.516807] Key type ._llcrypt registered [ 8184.517934] Key type .llcrypt registered [ 8184.558207] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 8184.979995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8185.034516] alg: No test for adler32 (adler32-zlib) [ 8185.883065] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 8185.961651] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 8185.963095] LNet: Accept secure, port 988 [ 8187.535159] Key type lgssc registered [ 8187.888813] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8190.407206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8192.201215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8194.091721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8195.818449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8201.140291] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 8204.351740] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 8207.968506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8210.235558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8212.294492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8214.398345] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8216.612100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8216.631505] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8216.635988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8217.717502] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8217.729071] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8217.764834] Lustre: lustre-MDT0000: new disk, initializing [ 8217.787851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8217.794201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8219.095405] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8223.242620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8223.268418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8223.302211] Lustre: 479271:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8223.325656] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8223.327255] Lustre: Skipped 1 previous similar message [ 8223.362276] Lustre: lustre-MDT0001: new disk, initializing [ 8223.394798] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8223.406264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8223.409037] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8224.741586] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8227.924243] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8229.090021] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8231.351119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8231.379029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8231.495067] Lustre: lustre-OST0000: new disk, initializing [ 8231.498100] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8231.535840] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8233.421916] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 8236.593619] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8237.062287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8237.065908] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8237.112046] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8238.018243] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8238.104907] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8239.504540] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8239.587865] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8242.148339] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8242.150541] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8242.151789] LustreError: Skipped 1 previous similar message [ 8242.158264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8242.161235] Lustre: Skipped 1 previous similar message [ 8246.317118] LustreError: 482176:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8246.396754] Lustre: server umount lustre-OST0000 complete [ 8249.311947] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8249.315536] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8249.321071] Lustre: Skipped 1 previous similar message [ 8249.323638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8252.384405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8252.384840] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8253.756097] LustreError: 482377:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8253.757914] LustreError: 482377:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8253.789288] Lustre: server umount lustre-MDT0000 complete [ 8255.029489] LustreError: 479263:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753448692 with bad export cookie 363227992172899120 [ 8255.034715] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8255.109888] LustreError: 482580:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8255.111892] LustreError: 482580:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8255.210284] Lustre: server umount lustre-MDT0001 complete [ 8263.335457] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 8266.289131] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 8270.394118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8273.174812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8275.309255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8277.927762] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8278.517673] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8133 sec ========== 09:05:15 (1753448715) [ 8279.136991] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 09:05:15 (1753448715) === [ 8280.355756] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 09:05:16 (1753448716) === [ 8293.842900] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 8294.849861] Key type lgssc unregistered [ 8294.985336] LNet: 487407:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8296.037344] LNet: Removed LNI 192.168.203.123@tcp [ 8296.340969] Key type .llcrypt unregistered [ 8296.343265] Key type ._llcrypt unregistered