[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 460128018 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003141] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.007998] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.008027] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010170] LSM: Security Framework initializing [ 0.011070] Yama: becoming mindful. [ 0.013048] SELinux: Initializing. [ 0.014066] *** VALIDATE selinux *** [ 0.022524] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027061] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028119] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030018] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031131] *** VALIDATE tmpfs *** [ 0.033391] *** VALIDATE proc *** [ 0.034208] *** VALIDATE cgroup *** [ 0.035007] *** VALIDATE cgroup2 *** [ 0.036273] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038007] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040024] Spectre V2 : User space: Vulnerable [ 0.041024] Speculative Store Bypass: Vulnerable [ 0.045000] debug: unmapping init [mem 0xffffffffb4059000-0xffffffffb4060fff] [ 0.046162] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047866] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048033] ... version: 2 [ 0.049015] ... bit width: 48 [ 0.050014] ... generic registers: 4 [ 0.051014] ... value mask: 0000ffffffffffff [ 0.052015] ... max period: 00007fffffffffff [ 0.053017] ... fixed-purpose events: 3 [ 0.054013] ... event mask: 000000070000000f [ 0.056244] rcu: Hierarchical SRCU implementation. [ 0.058909] smp: Bringing up secondary CPUs ... [ 0.059801] x86: Booting SMP configuration: [ 0.060040] .... node #0, CPUs: #1 #2 #3 [ 0.065267] smp: Brought up 1 node, 4 CPUs [ 0.067014] smpboot: Max logical packages: 1 [ 0.068023] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.230000] node 0 deferred pages initialised in 159ms [ 0.233014] devtmpfs: initialized [ 0.234307] x86/mm: Memory block size: 128MB [ 0.236799] gcov: version magic: 0x41383552 [ 0.238312] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.239085] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.240307] pinctrl core: initialized pinctrl subsystem [ 0.241208] [ 0.241670] ************************************************************* [ 0.242021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.243014] ** ** [ 0.244012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.245012] ** ** [ 0.246010] ** This means that this kernel is built to expose internal ** [ 0.247010] ** IOMMU data structures, which may compromise security on ** [ 0.248013] ** your system. ** [ 0.249012] ** ** [ 0.250016] ** If you see this message and you are not debugging the ** [ 0.251016] ** kernel, report this immediately to your vendor! ** [ 0.252013] ** ** [ 0.253015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.254015] ************************************************************* [ 0.255963] NET: Registered protocol family 16 [ 0.256498] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.257072] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.258092] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.259491] cpuidle: using governor menu [ 0.261703] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.262565] PCI: Using configuration type 1 for base access [ 0.263126] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.270162] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.271036] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.273120] cryptd: max_cpu_qlen set to 1000 [ 0.276183] ACPI: Added _OSI(Module Device) [ 0.277015] ACPI: Added _OSI(Processor Device) [ 0.278021] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.279018] ACPI: Added _OSI(Processor Aggregator Device) [ 0.282604] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.286085] ACPI: Interpreter enabled [ 0.287055] ACPI: PM: (supports S0 S3 S4 S5) [ 0.288016] ACPI: Using IOAPIC for interrupt routing [ 0.289138] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.290481] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.300072] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.301044] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.302024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.303097] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.305332] acpiphp: Slot [2] registered [ 0.306188] acpiphp: Slot [3] registered [ 0.307092] acpiphp: Slot [4] registered [ 0.308126] acpiphp: Slot [5] registered [ 0.309142] acpiphp: Slot [6] registered [ 0.310171] acpiphp: Slot [7] registered [ 0.311159] acpiphp: Slot [8] registered [ 0.312155] acpiphp: Slot [9] registered [ 0.313156] acpiphp: Slot [10] registered [ 0.314156] acpiphp: Slot [11] registered [ 0.315123] acpiphp: Slot [12] registered [ 0.316061] acpiphp: Slot [13] registered [ 0.317092] acpiphp: Slot [14] registered [ 0.318150] acpiphp: Slot [15] registered [ 0.319116] acpiphp: Slot [16] registered [ 0.320121] acpiphp: Slot [17] registered [ 0.321153] acpiphp: Slot [18] registered [ 0.322142] acpiphp: Slot [19] registered [ 0.323132] acpiphp: Slot [20] registered [ 0.324124] acpiphp: Slot [21] registered [ 0.325148] acpiphp: Slot [22] registered [ 0.326126] acpiphp: Slot [23] registered [ 0.327145] acpiphp: Slot [24] registered [ 0.328123] acpiphp: Slot [25] registered [ 0.329114] acpiphp: Slot [26] registered [ 0.330124] acpiphp: Slot [27] registered [ 0.331131] acpiphp: Slot [28] registered [ 0.332140] acpiphp: Slot [29] registered [ 0.333138] acpiphp: Slot [30] registered [ 0.334134] acpiphp: Slot [31] registered [ 0.335070] PCI host bridge to bus 0000:00 [ 0.336044] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.338031] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.340023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.342018] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.344016] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.346042] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.348249] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.352657] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.356572] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.365692] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.370730] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.372016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.374022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.376011] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.378269] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.380692] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.382034] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.384561] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.390013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.399000] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.401012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.405418] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.414017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.422027] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.446013] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.455561] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.468340] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.479026] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.503021] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.520972] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.528021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.534056] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.562025] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.576398] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.587018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.597021] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.623031] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.639023] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.647027] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.669061] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.696021] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.708000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.715019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.723017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.747039] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.762000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.765409] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.767345] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.769355] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.771214] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.775076] iommu: Default domain type: Passthrough [ 0.776713] SCSI subsystem initialized [ 0.779162] ACPI: bus type USB registered [ 0.780179] usbcore: registered new interface driver usbfs [ 0.783140] usbcore: registered new interface driver hub [ 0.786102] usbcore: registered new device driver usb [ 0.789240] pps_core: LinuxPPS API ver. 1 registered [ 0.792031] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.797087] PTP clock support registered [ 0.800061] EDAC MC: Ver: 3.0.0 [ 0.802179] PCI: Using ACPI for IRQ routing [ 0.803686] NetLabel: Initializing [ 0.804023] NetLabel: domain hash size = 128 [ 0.805000] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.808113] NetLabel: unlabeled traffic allowed by default [ 0.810381] vgaarb: loaded [ 0.812110] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.814014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.820521] clocksource: Switched to clocksource kvm-clock [ 0.933545] VFS: Disk quotas dquot_6.6.0 [ 0.935186] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.937724] *** VALIDATE ramfs *** [ 0.939049] *** VALIDATE hugetlbfs *** [ 0.940597] pnp: PnP ACPI init [ 0.942838] pnp: PnP ACPI: found 6 devices [ 0.961056] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.964192] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.965693] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.967640] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.969956] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.972154] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.974572] NET: Registered protocol family 2 [ 0.976875] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.981158] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.985171] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.990191] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.993530] TCP: Hash tables configured (established 65536 bind 65536) [ 0.996344] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.999079] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.001589] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.005184] NET: Registered protocol family 1 [ 1.008350] RPC: Registered named UNIX socket transport module. [ 1.009976] RPC: Registered udp transport module. [ 1.011715] RPC: Registered tcp transport module. [ 1.013098] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.015470] NET: Registered protocol family 44 [ 1.017104] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.019168] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.021441] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.023666] PCI: CLS 0 bytes, default 64 [ 1.026204] Unpacking initramfs... [ 2.422133] debug: unmapping init [mem 0xffff9c777cc54000-0xffff9c777ffbffff] [ 2.426098] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.428545] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.431529] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.952758] Initialise system trusted keyrings [ 2.954469] Key type blacklist registered [ 2.956413] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.972202] zbud: loaded [ 2.977849] *** VALIDATE nfs *** [ 2.978839] *** VALIDATE nfs4 *** [ 2.983085] pstore: using deflate compression [ 2.989968] Platform Keyring initialized [ 3.113251] NET: Registered protocol family 38 [ 3.115050] Key type asymmetric registered [ 3.116858] Asymmetric key parser 'x509' registered [ 3.119516] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.126398] io scheduler mq-deadline registered [ 3.128016] io scheduler kyber registered [ 3.130824] io scheduler bfq registered [ 3.133668] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.136827] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.139665] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.142536] ACPI: Power Button [PWRF] [ 3.263913] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.364264] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.563566] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.653935] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.875927] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.907526] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.935621] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.940659] Non-volatile memory driver v1.3 [ 3.942075] Linux agpgart interface v0.103 [ 3.992444] virtio_blk virtio1: [vda] 133800 512-byte logical blocks (68.5 MB/65.3 MiB) [ 3.995037] vda: detected capacity change from 0 to 68505600 [ 4.015394] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.019935] vdb: detected capacity change from 0 to 1073741824 [ 4.045952] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.049392] vdc: detected capacity change from 0 to 2621440000 [ 4.066762] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.071230] vdd: detected capacity change from 0 to 2621440000 [ 4.085829] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.089900] vde: detected capacity change from 0 to 4294967296 [ 4.103987] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.108144] vdf: detected capacity change from 0 to 4294967296 [ 4.116795] libphy: Fixed MDIO Bus: probed [ 4.123538] usbcore: registered new interface driver usbserial_generic [ 4.126944] usbserial: USB Serial support registered for generic [ 4.129671] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.135326] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.137785] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.140667] mousedev: PS/2 mouse device common for all mice [ 4.143504] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.145726] rtc_cmos 00:05: RTC can wake from S4 [ 4.150558] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.155629] rtc_cmos 00:05: registered as rtc0 [ 4.159642] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.160472] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.162834] intel_pstate: CPU model not supported [ 4.168970] hid: raw HID events driver (C) Jiri Kosina [ 4.171357] usbcore: registered new interface driver usbhid [ 4.173898] usbhid: USB HID core driver [ 4.175785] drop_monitor: Initializing network drop monitor service [ 4.178239] Initializing XFRM netlink socket [ 4.180555] NET: Registered protocol family 10 [ 4.183697] Segment Routing with IPv6 [ 4.185054] NET: Registered protocol family 17 [ 4.188355] mpls_gso: MPLS GSO support [ 4.195470] RAS: Correctable Errors collector initialized. [ 4.197155] AVX version of gcm_enc/dec engaged. [ 4.198351] AES CTR mode by8 optimization enabled [ 4.304813] sched_clock: Marking stable (4304638664, 0)->(5301656209, -997017545) [ 4.308652] registered taskstats version 1 [ 4.327589] Loading compiled-in X.509 certificates [ 4.332993] zswap: loaded using pool lzo/zbud [ 4.366526] Key type big_key registered [ 4.386593] Key type encrypted registered [ 4.389815] ima: No TPM chip found, activating TPM-bypass! [ 4.393282] ima: Allocated hash algorithm: sha1 [ 4.395485] ima: No architecture policies found [ 4.397165] evm: Initialising EVM extended attributes: [ 4.398473] evm: security.selinux [ 4.399678] evm: security.ima [ 4.400594] evm: security.capability [ 4.401665] evm: HMAC attrs: 0x1 [ 4.403758] rtc_cmos 00:05: setting system clock to 2025-10-07 04:55:14 UTC (1759812914) [ 4.413350] debug: unmapping init [mem 0xffffffffb5003000-0xffffffffb51fffff] [ 4.416572] debug: unmapping init [mem 0xffffffffb3d82000-0xffffffffb4058fff] [ 4.426777] Write protecting the kernel read-only data: 28672k [ 4.431681] debug: unmapping init [mem 0xffffffffb2403000-0xffffffffb25fffff] [ 4.436474] debug: unmapping init [mem 0xffffffffb2d14000-0xffffffffb2dfffff] [ 4.488878] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.496578] systemd[1]: Detected virtualization kvm. [ 4.497781] systemd[1]: Detected architecture x86-64. [ 4.499399] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.531198] systemd[1]: No hostname configured. [ 4.533780] systemd[1]: Set hostname to . [ 4.535841] random: systemd: uninitialized urandom read (16 bytes read) [ 4.538953] systemd[1]: Initializing machine ID from random generator. [ 4.749801] random: systemd: uninitialized urandom read (16 bytes read) [ 4.755645] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 4.762466] random: systemd: uninitialized urandom read (16 bytes read) [ 4.765297] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.773319] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... Starting Journal Service... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.725631] device-mapper: uevent: version 1.0.3 [ 5.728108] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 6.920224] virtio_net virtio0 ens2: renamed from eth0 [ 6.958019] random: fast init done [ 7.120574] scsi host0: ata_piix [ 7.134378] scsi host1: ata_piix [ 7.135995] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.143790] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 11.922478] random: crng init done [ 11.929163] random: 7 urandom warning(s) missed due to ratelimiting [ 14.325274] dracut-initqueue[589]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 15.169434] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 16.942847] printk: systemd: 26 output lines suppressed due to ratelimiting [ 17.335966] SELinux: Disabled at runtime. [ 17.432917] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 17.442337] systemd[1]: Detected virtualization kvm. [ 17.445373] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 18.262939] systemd[1]: initrd-switch-root.service: Succeeded. [ 18.267743] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 18.276130] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 18.280644] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 18.283330] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 18.290685] systemd[1]: Starting Journal Service... Starting Journal Service... [ 18.304370] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... Mounting POSIX Message Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on RPCbind Server Activation Socket. Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ 18.668732] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 19.110562] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 19.808468] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 19.896916] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 20.221328] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 20.277316] EDAC sbridge: Ver: 1.1.2 [ 23.581742] Key type dns_resolver registered [ 23.964475] NFS: Registering the id_resolver key type [ 23.970908] Key type id_resolver registered [ 23.973336] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... Starting Login Service... Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg353-server login: [ 56.119272] libcfs: loading out-of-tree module taints kernel. [ 56.132941] Key type ._llcrypt registered [ 56.134186] Key type .llcrypt registered [ 56.185418] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 64.911857] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 65.607547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 65.618281] alg: No test for adler32 (adler32-zlib) [ 66.680088] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 67.060549] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 68.688158] Key type lgssc registered [ 69.363569] Lustre: Echo OBD driver; http://www.lustre.org/ [ 77.523570] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 78.694826] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 82.850311] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 86.066189] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 89.248143] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 96.074356] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 101.720079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 101.744868] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 101.758362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 102.861401] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 102.877844] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 102.920506] Lustre: lustre-MDT0000: new disk, initializing [ 102.954567] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 102.962811] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 104.487462] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 110.191051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 110.224562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 110.268262] Lustre: 6486:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 110.286799] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 110.288858] Lustre: Skipped 1 previous similar message [ 110.332988] Lustre: lustre-MDT0001: new disk, initializing [ 110.372363] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 110.384659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 110.388936] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 111.947155] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 114.543818] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 118.266532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 118.305460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 118.407349] Lustre: lustre-OST0000: new disk, initializing [ 118.409597] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 118.427644] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 118.525593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 118.529419] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 118.570097] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 120.640636] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 127.367362] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 127.404797] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 127.458257] Lustre: lustre-OST0001: new disk, initializing [ 127.460671] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 127.484684] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 129.896629] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 136.406889] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 136.686316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 136.688900] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 136.725066] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 140.338666] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 147.700229] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing check_logdir /tmp/testlogs/ [ 149.642309] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing yml_node [ 151.746479] Lustre: DEBUG MARKER: Client: 2.16.58.105 [ 152.881887] Lustre: DEBUG MARKER: MDS: 2.16.58.105 [ 153.958434] Lustre: DEBUG MARKER: OSS: 2.16.58.105 [ 154.745960] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Oct 7 00:57:44 EDT 2025 [ 163.122996] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 163.752825] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 177.120513] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 177.122968] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 177.127348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 177.632927] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 177.633534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 177.638748] Lustre: Skipped 2 previous similar messages [ 177.641568] Lustre: Skipped 1 previous similar message [ 181.884902] LustreError: 12186:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 181.918224] Lustre: server umount lustre-MDT0000 complete [ 183.243654] LustreError: 6476:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759813093 with bad export cookie 14874041806612591162 [ 183.245136] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 183.248124] LustreError: 6476:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 187.872806] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 187.872920] LustreError: 6491:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 187.873436] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 187.873441] Lustre: Skipped 1 previous similar message [ 187.877127] Lustre: Skipped 1 previous similar message [ 187.883779] LustreError: 6491:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 188.896832] LustreError: 6491:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 188.905359] LustreError: 6491:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 189.373644] LustreError: 12388:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 189.376087] LustreError: 12388:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 189.466228] Lustre: server umount lustre-MDT0001 complete [ 191.086818] LustreError: 12589:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 191.089473] LustreError: 12589:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 191.113316] Lustre: server umount lustre-OST0000 complete [ 192.917385] Lustre: server umount lustre-OST0001 complete [ 195.485990] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 199.201820] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 204.179325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 208.384661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 214.645924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 221.199713] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 228.219551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 228.306081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 228.644553] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 228.684705] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 228.757271] Lustre: lustre-MDT0000: new disk, initializing [ 228.841775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 228.860503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 233.168375] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 244.164849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 244.246517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 244.360886] Lustre: 14970:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 244.413434] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 244.418803] Lustre: Skipped 1 previous similar message [ 244.554064] Lustre: lustre-MDT0001: new disk, initializing [ 244.695441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 244.747265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 244.756585] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 249.007426] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 257.130450] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 261.553590] hrtimer: interrupt took 4616263 ns [ 261.644196] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 268.376847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 268.445609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 268.696942] Lustre: lustre-OST0000: new disk, initializing [ 268.699864] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 268.805705] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 270.653556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 270.660513] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 270.760513] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 273.154484] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 278.964157] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 282.057372] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 282.189447] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 284.954687] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 285.149402] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 291.298716] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 291.300855] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 291.313961] LustreError: Skipped 1 previous similar message [ 291.319489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 291.321935] Lustre: Skipped 1 previous similar message [ 291.338291] Lustre: Skipped 4 previous similar messages [ 292.589116] LustreError: 17909:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 292.591570] LustreError: 17909:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 292.698331] Lustre: server umount lustre-OST0000 complete [ 295.907092] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 295.914587] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 295.921100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 301.204788] LustreError: 18113:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 301.210588] LustreError: 18113:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 301.306912] Lustre: server umount lustre-MDT0000 complete [ 301.537101] LustreError: 14983:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 301.545315] LustreError: 14983:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 303.603188] LustreError: 17910:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759813213 with bad export cookie 14874041806612592625 [ 303.606530] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 303.610632] LustreError: 17910:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 303.839758] Lustre: server umount lustre-MDT0001 complete [ 306.301311] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 01:00:16 (1759813216) [ 307.788196] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 308.954341] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 01:00:18 (1759813218) [ 313.009167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 313.249433] LustreError: 18983:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 313.307050] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 315.615722] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 318.434110] LustreError: 18983:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 320.948567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 323.807329] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 327.713766] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 330.252836] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 335.177299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 339.638043] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 343.632427] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 352.226158] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 352.227590] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 352.240128] Lustre: Skipped 1 previous similar message [ 352.245613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 352.249153] Lustre: Skipped 3 previous similar messages [ 353.978217] LustreError: 21244:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 353.985569] LustreError: 21244:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 354.115983] Lustre: server umount lustre-OST0000 complete [ 357.347354] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 371.168295] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 371.225024] LustreError: 21445:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 371.229769] LustreError: 21445:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 371.294358] Lustre: server umount lustre-MDT0000 complete [ 372.705341] LustreError: 19694:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 372.716981] LustreError: 19694:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 373.548312] LustreError: 18963:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759813283 with bad export cookie 14874041806612593507 [ 373.553989] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 373.798607] Lustre: server umount lustre-MDT0001 complete [ 380.519460] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 382.251914] Key type lgssc unregistered [ 382.426613] LNet: 22180:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 382.432905] LNetError: 22180:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 383.462389] LNet: Removed LNI 192.168.203.153@tcp [ 384.072118] Key type .llcrypt unregistered [ 384.074181] Key type ._llcrypt unregistered [ 397.313453] Key type ._llcrypt registered [ 397.315025] Key type .llcrypt registered [ 397.376118] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 397.883496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 397.897962] alg: No test for adler32 (adler32-zlib) [ 398.819431] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 398.930523] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 400.544155] Key type lgssc registered [ 401.023976] Lustre: Echo OBD driver; http://www.lustre.org/ [ 405.771042] loop: module loaded [ 405.845577] loop0: detected capacity change from 0 to 204800000 [ 405.980354] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 405.987641] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 406.202791] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 406.209930] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 406.245834] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 406.274885] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 406.300824] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 406.348895] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 410.113799] loop0: detected capacity change from 0 to 204800000 [ 410.230580] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 410.236395] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 410.251727] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 410.353900] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 413.356054] loop0: detected capacity change from 0 to 204800000 [ 416.960551] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 416.996958] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 417.009652] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 418.152023] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 418.188531] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 418.243479] Lustre: lustre-MDT0000: new disk, initializing [ 418.302243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 418.314253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 420.815893] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 426.145889] loop1: detected capacity change from 0 to 204800000 [ 429.642597] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 429.699358] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 429.882890] Lustre: lustre-OST07c6: new disk, initializing [ 429.889795] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 429.892583] Lustre: Skipped 1 previous similar message [ 429.948887] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 432.113217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 432.126281] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 432.157226] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 433.619319] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 524.258913] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 524.259158] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 524.267122] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 529.376988] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 529.518069] LustreError: 26521:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 529.574719] Lustre: server umount lustre-OST07c6 complete [ 532.945679] LustreError: 27022:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 532.948548] LustreError: 27022:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 533.039575] Lustre: server umount lustre-MDT0000 complete [ 539.579426] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 540.767591] Key type lgssc unregistered [ 540.902446] LNet: 27854:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 540.905646] LNetError: 27854:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 540.914374] LNet: Removed LNI 192.168.203.153@tcp [ 541.229161] Key type .llcrypt unregistered [ 541.231065] Key type ._llcrypt unregistered [ 549.072665] Key type ._llcrypt registered [ 549.073765] Key type .llcrypt registered [ 549.115278] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 549.453529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 549.495796] alg: No test for adler32 (adler32-zlib) [ 550.361207] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 550.443233] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 552.024152] Key type lgssc registered [ 552.421600] Lustre: Echo OBD driver; http://www.lustre.org/ [ 555.440374] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 555.451090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 556.634717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 558.127655] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 561.405564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 561.554404] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 562.989146] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 564.903543] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 566.300473] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 568.916305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 569.047068] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 571.287684] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 573.175987] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 581.350166] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 585.600221] LustreError: 31300:0:(qsd_reint.c:618:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:2 until pending updates are flushed. [ 590.819125] LustreError: lustre-OST0000-osc-MDT0000: operation ost_setattr to node 0@lo failed: rc = -107 [ 590.819211] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 590.822145] LustreError: Skipped 1 previous similar message [ 590.829623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 591.844857] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 591.846108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 591.848384] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 594.617890] LustreError: 31552:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 594.709050] Lustre: server umount lustre-OST0000 complete [ 596.961751] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 596.962524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 596.967318] Lustre: Skipped 1 previous similar message [ 596.971259] Lustre: Skipped 2 previous similar messages [ 602.082027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 602.085168] Lustre: Skipped 1 previous similar message [ 602.367633] LustreError: 31753:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 602.370577] LustreError: 31753:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 602.401326] Lustre: server umount lustre-MDT0000 complete [ 603.876586] LustreError: 29064:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759813513 with bad export cookie 11673523996920019269 [ 603.879420] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 603.883458] LustreError: 29064:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 603.952219] LustreError: 31955:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 603.955445] LustreError: 31955:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 604.038073] Lustre: server umount lustre-MDT0001 complete [ 608.017684] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 609.267803] Key type lgssc unregistered [ 609.421515] LNet: 32485:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 609.424853] LNetError: 32485:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 609.436369] LNet: Removed LNI 192.168.203.153@tcp [ 609.792401] Key type .llcrypt unregistered [ 609.793881] Key type ._llcrypt unregistered [ 618.269915] Key type ._llcrypt registered [ 618.271165] Key type .llcrypt registered [ 618.325529] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 618.709434] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 618.781268] alg: No test for adler32 (adler32-zlib) [ 619.674367] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 619.794371] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 621.408330] Key type lgssc registered [ 621.873944] Lustre: Echo OBD driver; http://www.lustre.org/ [ 624.955300] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 624.967231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 626.181157] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 627.752360] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 631.078874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 631.225457] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 632.742570] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 634.888224] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 636.377683] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 639.315701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 639.470248] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 641.796763] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 643.969901] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 651.755612] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 661.985740] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 661.986490] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 661.988122] LustreError: Skipped 1 previous similar message [ 661.992538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 661.995810] Lustre: Skipped 1 previous similar message [ 664.302115] LustreError: 35983:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 664.368528] Lustre: server umount lustre-OST0000 complete [ 667.106249] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 667.106355] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 667.111812] Lustre: Skipped 1 previous similar message [ 667.117036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 667.119723] Lustre: Skipped 1 previous similar message [ 671.935207] LustreError: 36186:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 671.938209] LustreError: 36186:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 671.965768] Lustre: server umount lustre-MDT0000 complete [ 672.225405] LustreError: 33731:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 672.230466] LustreError: 33731:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 673.400538] LustreError: 33694:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759813583 with bad export cookie 8177980146547411688 [ 673.404127] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 673.405059] LustreError: 33694:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 673.469131] LustreError: 36387:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 673.471941] LustreError: 36387:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 673.562814] Lustre: server umount lustre-MDT0001 complete [ 677.526716] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 678.672681] Key type lgssc unregistered [ 678.806408] LNet: 36919:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 678.809661] LNetError: 36919:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 678.821261] LNet: Removed LNI 192.168.203.153@tcp [ 679.120729] Key type .llcrypt unregistered [ 679.121902] Key type ._llcrypt unregistered [ 688.175429] Key type ._llcrypt registered [ 688.176508] Key type .llcrypt registered [ 688.221961] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 688.584701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 688.609561] alg: No test for adler32 (adler32-zlib) [ 689.473896] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 689.561866] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 691.144139] Key type lgssc registered [ 691.504223] Lustre: Echo OBD driver; http://www.lustre.org/ [ 694.216752] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 694.224210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 695.393629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 696.719830] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 699.628332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 699.771892] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 701.222206] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 703.150811] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 704.397549] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 706.815842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 706.935939] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 708.963841] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 710.924462] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 720.230926] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 725.475769] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 725.476030] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 725.476654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 725.480777] Lustre: Skipped 1 previous similar message [ 726.528325] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.53@tcp (stopping) [ 726.531524] Lustre: Skipped 2 previous similar messages [ 729.855244] LustreError: 40560:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 729.886087] Lustre: server umount lustre-MDT0000 complete [ 730.594126] LustreError: 38292:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 730.599945] LustreError: 38292:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 731.184240] LustreError: 39912:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759813641 with bad export cookie 9740645268299836745 [ 731.187704] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 731.188543] LustreError: 39912:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 731.641034] LustreError: 39004:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 731.650636] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.53@tcp (stopping) [ 735.713186] LustreError: 38289:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 735.713587] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 735.725776] Lustre: Skipped 1 previous similar message [ 735.728518] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 737.343538] LustreError: 40763:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 737.345398] LustreError: 40763:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 737.434891] Lustre: server umount lustre-MDT0001 complete [ 755.436312] LustreError: 40986:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 755.438604] LustreError: 40986:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 755.503636] Lustre: server umount lustre-OST0000 complete [ 761.201820] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 762.278254] Key type lgssc unregistered [ 762.417440] LNet: 41822:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 762.419807] LNetError: 41822:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 762.428430] LNet: Removed LNI 192.168.203.153@tcp [ 762.749377] Key type .llcrypt unregistered [ 762.751136] Key type ._llcrypt unregistered [ 771.691700] Key type ._llcrypt registered [ 771.692680] Key type .llcrypt registered [ 771.728240] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 772.038842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 772.071673] alg: No test for adler32 (adler32-zlib) [ 772.920068] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 773.005682] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 774.584143] Key type lgssc registered [ 774.925188] Lustre: Echo OBD driver; http://www.lustre.org/ [ 777.502706] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 777.511145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 778.675935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 780.084140] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 782.991480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 783.121431] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 784.442835] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 786.303946] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 787.580439] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 789.893302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 790.017145] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 791.938771] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 793.772142] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 803.364794] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 808.419569] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 808.419755] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 808.423651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 808.425473] LustreError: Skipped 1 previous similar message [ 808.427161] Lustre: Skipped 1 previous similar message [ 809.473544] Lustre: lustre-OST0000: Not available for connect from 192.168.203.53@tcp (stopping) [ 812.591100] LustreError: 45461:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 812.653096] Lustre: server umount lustre-OST0000 complete [ 825.455196] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 825.455990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 825.459591] Lustre: Skipped 2 previous similar messages [ 825.462944] Lustre: Skipped 1 previous similar message [ 828.513909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 828.516249] Lustre: Skipped 1 previous similar message [ 831.611155] LustreError: 45838:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 831.613034] LustreError: 45838:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 831.636460] Lustre: server umount lustre-MDT0000 complete [ 832.878078] LustreError: 43177:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759813742 with bad export cookie 442572126104102718 [ 832.881405] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 832.882262] LustreError: 43177:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 847.328212] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 847.359095] LustreError: 46039:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 847.361332] LustreError: 46039:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 847.435451] Lustre: server umount lustre-MDT0001 complete [ 850.897759] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 852.020254] Key type lgssc unregistered [ 852.152292] LNet: 46571:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 852.155676] LNetError: 46571:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 852.164347] LNet: Removed LNI 192.168.203.153@tcp [ 852.486234] Key type .llcrypt unregistered [ 852.487166] Key type ._llcrypt unregistered [ 861.027018] Key type ._llcrypt registered [ 861.028431] Key type .llcrypt registered [ 861.066018] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 861.361725] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 861.369636] alg: No test for adler32 (adler32-zlib) [ 862.220911] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 862.292913] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 863.872121] Key type lgssc registered [ 864.188635] Lustre: Echo OBD driver; http://www.lustre.org/ [ 866.643054] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 866.650280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 867.794361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 869.050716] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 871.993247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 872.174758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 873.640778] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 875.515301] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 876.803952] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 879.245106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 879.368150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 881.388888] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 883.263317] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 892.710843] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 896.128518] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 897.373455] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 908.116573] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2025-10-07 1h10m17s [ 908.678354] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 913.378361] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 913.380390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 913.381909] Lustre: Skipped 2 previous similar messages [ 913.383918] Lustre: Skipped 1 previous similar message [ 913.387664] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 127.0.0.2@tcp failed: rc = -107 [ 914.402260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 914.405210] Lustre: Skipped 2 previous similar messages [ 915.447510] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.53@tcp (stopping) [ 915.450679] Lustre: Skipped 1 previous similar message [ 915.518562] LustreError: 50534:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 915.545842] Lustre: server umount lustre-MDT0000 complete [ 916.638840] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 918.837603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 918.875612] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 918.963449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 918.993513] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 919.974804] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 920.357786] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 921.535124] Lustre: DEBUG MARKER: Wait for df (35969) ... [ 922.125688] Lustre: DEBUG MARKER: done [ 923.616952] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 923.619683] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 923.625343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 925.229918] LustreError: 51582:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 925.232709] LustreError: 51582:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 925.290944] Lustre: server umount lustre-OST0000 complete [ 928.574327] LustreError: 51785:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 928.577038] LustreError: 51785:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 928.608430] Lustre: server umount lustre-MDT0000 complete [ 930.273640] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 930.273686] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 930.276678] LustreError: Skipped 1 previous similar message [ 930.279788] Lustre: Skipped 1 previous similar message [ 930.283400] LustreError: 47947:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 930.287845] LustreError: 47947:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 931.297532] LustreError: 47947:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 931.302807] LustreError: 47947:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 936.353111] LustreError: 47947:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 936.359022] LustreError: 47947:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 942.561608] LustreError: 47947:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 942.567427] LustreError: 47947:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 944.096202] Lustre: 47938:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759813838/real 1759813838] req@0000000094de0735 x1845298485828608/t0(0) o101->MGC192.168.203.153@tcp@0@lo:26/25 lens 328/344 e 0 to 1 dl 1759813854 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 944.106266] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 944.111804] LustreError: 47938:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 944.176694] LustreError: 51985:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 944.178934] LustreError: 51985:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 944.263218] Lustre: server umount lustre-MDT0001 complete [ 948.385489] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 949.467936] Key type lgssc unregistered [ 949.597440] LNet: 52518:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 949.599598] LNetError: 52518:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 949.610347] LNet: Removed LNI 192.168.203.153@tcp [ 949.944285] Key type .llcrypt unregistered [ 949.945860] Key type ._llcrypt unregistered [ 961.275591] Key type ._llcrypt registered [ 961.276911] Key type .llcrypt registered [ 961.315199] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 967.386165] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 967.674333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 967.681528] alg: No test for adler32 (adler32-zlib) [ 968.537180] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 968.615742] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 970.192145] Key type lgssc registered [ 970.561912] Lustre: Echo OBD driver; http://www.lustre.org/ [ 973.482086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 975.858759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 977.832688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 979.917838] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 982.224108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 982.242326] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 982.248789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 983.335406] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 983.347446] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 983.387203] Lustre: lustre-MDT0000: new disk, initializing [ 983.417179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 983.423991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 984.759356] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 988.875786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 988.900418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 988.934048] Lustre: 56439:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 988.956227] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 988.958491] Lustre: Skipped 1 previous similar message [ 988.992117] Lustre: lustre-MDT0001: new disk, initializing [ 989.023417] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 989.035093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 989.038102] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 990.353923] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 993.365275] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 994.521353] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 996.745043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 996.769635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 996.862118] Lustre: lustre-OST0000: new disk, initializing [ 996.863968] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 996.890752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 998.732628] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1000.637926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1000.641783] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1000.679140] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1001.809334] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1003.178382] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1003.250892] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1004.555477] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1004.631686] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1006.050362] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1006.050975] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1006.052970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1006.055813] Lustre: Skipped 1 previous similar message [ 1006.058817] LustreError: Skipped 1 previous similar message [ 1007.344095] LustreError: 59376:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1007.412536] Lustre: server umount lustre-OST0000 complete [ 1009.632827] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1009.635117] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1009.640351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1009.642509] Lustre: Skipped 1 previous similar message [ 1011.169238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1011.169963] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1014.846795] LustreError: 59577:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1014.849046] LustreError: 59577:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1014.881883] Lustre: server umount lustre-MDT0000 complete [ 1016.194055] LustreError: 56431:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759813926 with bad export cookie 3868677777540610027 [ 1016.196856] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1016.197979] LustreError: 56431:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1016.262915] LustreError: 59778:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1016.264522] LustreError: 59778:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1016.339237] Lustre: server umount lustre-MDT0001 complete [ 1019.060413] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 01:12:09 (1759813929) [ 1021.346732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1021.510661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1022.782775] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1025.611024] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1025.724333] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1027.033676] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1028.791508] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1029.924841] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1032.067139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1034.034899] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1035.855351] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1039.199392] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2025-10-07 1h12m29s [ 1039.722099] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1042.090107] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1042.911161] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1042.925910] LustreError: 60442:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1063.428590] Lustre: lustre-MDT0000: Client 6eeeab42-a9be-4931-9fab-d6577c24bc91 (at 192.168.203.53@tcp) reconnecting [ 1082.968112] LustreError: 60442:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1083.678744] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1084.451899] Lustre: DEBUG MARKER: done [ 1086.434668] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1086.435239] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1086.438571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1086.440524] LustreError: Skipped 1 previous similar message [ 1086.443402] Lustre: Skipped 2 previous similar messages [ 1091.553962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1091.556125] Lustre: Skipped 1 previous similar message [ 1092.079087] LustreError: 63451:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1092.081177] LustreError: 63451:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1092.142531] Lustre: server umount lustre-OST0000 complete [ 1096.674188] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1096.678562] Lustre: Skipped 2 previous similar messages [ 1099.582632] LustreError: 63652:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1099.585237] LustreError: 63652:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1099.611403] Lustre: server umount lustre-MDT0000 complete [ 1100.929267] LustreError: 62065:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759814011 with bad export cookie 3868677777540610895 [ 1100.932920] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1100.933782] LustreError: 62065:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1101.084690] Lustre: server umount lustre-MDT0001 complete [ 1104.870927] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 1105.999116] Key type lgssc unregistered [ 1106.134386] LNet: 64384:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1106.138129] LNetError: 64384:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1106.146388] LNet: Removed LNI 192.168.203.153@tcp [ 1106.446590] Key type .llcrypt unregistered [ 1106.447728] Key type ._llcrypt unregistered [ 1117.488026] Key type ._llcrypt registered [ 1117.489251] Key type .llcrypt registered [ 1117.531443] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 1123.964123] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 1124.448060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1124.455914] alg: No test for adler32 (adler32-zlib) [ 1125.321131] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 1125.407404] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 1126.992163] Key type lgssc registered [ 1127.420806] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1130.690798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1133.096657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1135.268431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1137.536605] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1139.957488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1139.977486] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1139.985831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1141.074167] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1141.087098] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1141.124824] Lustre: lustre-MDT0000: new disk, initializing [ 1141.152431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1141.160820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1142.567631] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1147.148113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1147.175626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1147.215739] Lustre: 68307:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1147.243311] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1147.246816] Lustre: Skipped 1 previous similar message [ 1147.286982] Lustre: lustre-MDT0001: new disk, initializing [ 1147.324703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1147.339888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1147.343527] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1148.761841] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1152.063442] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1153.356735] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1155.673357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1155.700536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1155.797593] Lustre: lustre-OST0000: new disk, initializing [ 1155.799525] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1155.830946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1157.866212] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1161.126542] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1162.240415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1162.244131] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1162.287340] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1162.655874] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1162.750676] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1164.205733] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1164.292109] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1167.331651] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1167.331895] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1167.337532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1167.339461] LustreError: Skipped 1 previous similar message [ 1171.117090] LustreError: 71243:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1171.184085] Lustre: server umount lustre-OST0000 complete [ 1172.960978] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1172.965067] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1172.970329] Lustre: Skipped 1 previous similar message [ 1172.973079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1172.975193] Lustre: Skipped 1 previous similar message [ 1177.569191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1177.569657] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1178.622667] LustreError: 71445:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1178.624588] LustreError: 71445:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1178.657081] Lustre: server umount lustre-MDT0000 complete [ 1179.945388] LustreError: 68298:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759814090 with bad export cookie 9762361784086873758 [ 1179.947891] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1179.948703] LustreError: 68298:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1180.010184] LustreError: 71647:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1180.012727] LustreError: 71647:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1180.091274] Lustre: server umount lustre-MDT0001 complete [ 1182.943231] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 01:14:52 (1759814092) [ 1183.509098] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1184.095684] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 01:14:54 (1759814094) [ 1188.290296] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 1189.869934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1190.057378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1190.412298] Lustre: Failing over lustre-MDT0000 [ 1190.414029] LustreError: 72722:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1190.417543] LustreError: 72722:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1190.420046] LustreError: 72722:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 0, retries 0, failed: rc = -5 [ 1190.482319] LustreError: 72774:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1190.484219] LustreError: 72774:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1190.557489] Lustre: server umount lustre-MDT0000 complete [ 1192.849362] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 01:15:02 (1759814102) [ 1195.254677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1195.409506] LustreError: 73342:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1195.417774] LustreError: 73342:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1195.428969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1196.855158] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1199.929440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1201.583719] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1203.645317] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1204.907391] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1207.192960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1207.317901] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1207.320985] Lustre: Skipped 1 previous similar message [ 1209.234541] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1210.929089] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1223.765713] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1225.697886] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1225.698299] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1225.699508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1225.699513] Lustre: Skipped 1 previous similar message [ 1225.701203] Lustre: Skipped 1 previous similar message [ 1230.817665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1230.819651] Lustre: Skipped 4 previous similar messages [ 1231.611702] LustreError: 75707:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1231.613539] LustreError: 75707:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1231.640614] Lustre: server umount lustre-MDT0000 complete [ 1232.871231] LustreError: 75573:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759814142 with bad export cookie 9762361784086874913 [ 1232.874533] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1232.875037] LustreError: 75573:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1233.032442] Lustre: server umount lustre-MDT0001 complete [ 1234.191368] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1235.723344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1238.532678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1242.080648] LustreError: 66520:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000001c26a83e x1845298761623168/t0(0) o250->MGC192.168.203.153@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1242.194145] LustreError: 74947:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1242.222938] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1242.249770] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1243.545870] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1246.305558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1246.391020] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1246.396452] Lustre: Skipped 1 previous similar message [ 1247.701988] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1249.411270] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1250.509954] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1251.812950] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1252.832233] Lustre: 66523:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759814146/real 1759814146] req@000000000073e337 x1845298761622784/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1759814162 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1254.880146] Lustre: 66524:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759814148/real 1759814148] req@0000000005e9cffb x1845298761622912/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1759814164 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1255.908464] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1256.928248] Lustre: 66521:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759814151/real 1759814151] req@0000000090a1c8ed x1845298761623040/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1759814167 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1261.026001] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1261.027702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1261.030013] Lustre: Skipped 2 previous similar messages [ 1261.033688] Lustre: Skipped 2 previous similar messages [ 1263.550312] LustreError: 78170:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1263.552265] LustreError: 78170:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1263.580853] Lustre: server umount lustre-MDT0000 complete [ 1264.861126] LustreError: 78084:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759814174 with bad export cookie 9762361784086876803 [ 1264.864463] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1264.866354] LustreError: 78084:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1265.008117] Lustre: server umount lustre-MDT0001 complete [ 1267.174547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1270.250223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1274.272571] LustreError: 66520:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000065e215fa x1845298761642752/t0(0) o250->MGC192.168.203.153@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1274.376599] LustreError: 74946:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1274.400899] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1274.403901] Lustre: Skipped 1 previous similar message [ 1274.427486] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1275.736537] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1278.688841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1278.776083] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1280.139975] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1281.886552] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1281.894558] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1282.912140] Lustre: 66522:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759814176/real 1759814176] req@00000000ed53d840 x1845298761642368/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1759814192 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1282.920547] Lustre: 66522:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1283.080088] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1287.139628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1287.141764] Lustre: Skipped 2 previous similar messages [ 1288.160133] Lustre: 66524:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759814181/real 1759814181] req@000000006bd0cd3d x1845298761642624/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1759814197 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1288.166496] Lustre: 66524:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1290.877553] LustreError: 80625:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1290.879750] LustreError: 80625:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1290.907926] Lustre: server umount lustre-MDT0000 complete [ 1292.147069] LustreError: 73321:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759814202 with bad export cookie 9762361784086878175 [ 1292.149596] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1292.151186] LustreError: 73321:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1292.276197] Lustre: server umount lustre-MDT0001 complete [ 1294.327949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1295.420625] Lustre: DEBUG MARKER: files compared the same [ 1306.418343] Lustre: server umount lustre-OST0000 complete [ 1311.817520] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 1312.926855] Key type lgssc unregistered [ 1313.065383] LNet: 82255:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1313.067626] LNetError: 82255:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1313.075304] LNet: Removed LNI 192.168.203.153@tcp [ 1313.382646] Key type .llcrypt unregistered [ 1313.383814] Key type ._llcrypt unregistered [ 1322.093141] Key type ._llcrypt registered [ 1322.094397] Key type .llcrypt registered [ 1322.126959] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 1322.401467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1322.408879] alg: No test for adler32 (adler32-zlib) [ 1323.249291] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 1323.321820] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 1324.896131] Key type lgssc registered [ 1325.233956] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1327.785536] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1327.791808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1328.942417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1330.189914] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug +malloc all [ 1332.836196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1332.928144] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1334.162118] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug +malloc all [ 1335.922483] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1337.156078] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1339.379560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1339.473453] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1341.296659] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug +malloc all [ 1343.025123] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1354.850228] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1364.448580] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1364.451502] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1364.456083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1365.472690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1365.473101] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1365.473454] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1367.276066] LustreError: 85895:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1367.331380] Lustre: server umount lustre-OST0000 complete [ 1369.056547] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1369.058659] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1369.064554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1369.066554] Lustre: Skipped 1 previous similar message [ 1374.654851] LustreError: 86096:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1374.657326] LustreError: 86096:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1374.680825] Lustre: server umount lustre-MDT0000 complete [ 1375.712591] LustreError: 85785:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1375.717903] LustreError: 85785:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1375.954617] LustreError: 83611:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759814286 with bad export cookie 11610128229306156830 [ 1375.955703] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1375.957537] LustreError: 83611:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1376.024313] LustreError: 86300:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1376.026057] LustreError: 86300:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1376.094646] Lustre: server umount lustre-MDT0001 complete [ 1379.718650] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 1380.763930] Key type lgssc unregistered [ 1380.891379] LNet: 86831:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1380.894268] LNetError: 86831:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1380.902306] LNet: Removed LNI 192.168.203.153@tcp [ 1381.207785] Key type .llcrypt unregistered [ 1381.209693] Key type ._llcrypt unregistered [ 1390.616656] Key type ._llcrypt registered [ 1390.617958] Key type .llcrypt registered [ 1390.655879] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 1390.986028] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1391.030237] alg: No test for adler32 (adler32-zlib) [ 1391.876430] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 1391.965815] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 1393.552132] Key type lgssc registered [ 1393.934583] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1396.447433] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1396.452809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1411.872306] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1416.992169] LustreError: 88177:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.153@tcp: failed processing log, type 4: rc = -110 [ 1443.616197] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1445.433126] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1447.176601] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1449.616836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1449.654378] Lustre: *** cfs_fail_loc=706, val=0*** [ 1449.783106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1449.807206] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1451.116931] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1454.046332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1454.182311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1455.576106] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1457.418567] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1458.695529] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1460.192921] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1460.195132] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1460.200134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1463.778803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1463.779370] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1465.583087] LustreError: 90497:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 1465.647957] Lustre: server umount lustre-OST0000 complete [ 1468.897554] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1468.901592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1468.904402] Lustre: Skipped 2 previous similar messages [ 1473.083085] LustreError: 90701:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1473.086086] LustreError: 90701:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1473.111887] Lustre: server umount lustre-MDT0000 complete [ 1474.017788] LustreError: 89153:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1474.022439] LustreError: 89153:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1474.266086] LustreError: 88184:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759814384 with bad export cookie 12906195080260646770 [ 1474.269459] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1474.269950] LustreError: 88184:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1474.328557] LustreError: 90902:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1474.330561] LustreError: 90902:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1474.401412] Lustre: server umount lustre-MDT0001 complete [ 1477.851838] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 1478.920757] Key type lgssc unregistered [ 1479.050427] LNet: 91434:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1479.053514] LNetError: 91434:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1479.061409] LNet: Removed LNI 192.168.203.153@tcp [ 1479.389469] Key type .llcrypt unregistered [ 1479.390447] Key type ._llcrypt unregistered [ 1488.195562] Key type ._llcrypt registered [ 1488.197278] Key type .llcrypt registered [ 1488.231699] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 1488.625753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1488.633569] alg: No test for adler32 (adler32-zlib) [ 1489.482824] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 1489.561738] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 1491.136139] Key type lgssc registered [ 1491.464084] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1493.821083] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1493.827614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1496.009687] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1498.421844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1498.589438] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1499.863574] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1502.649862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1502.756135] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1504.478776] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1507.127020] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1507.130905] Lustre: MGS: Client 6115783f-48d4-4318-8b3c-89acf518d23d (at 0@lo) reconnecting [ 1507.211511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1507.239884] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1508.424408] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1517.537546] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1517.540714] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1517.546055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1518.565473] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1518.566510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1518.566611] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1518.573459] Lustre: Skipped 1 previous similar message [ 1522.284192] LustreError: 95442:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1522.339522] Lustre: server umount lustre-OST0000 complete [ 1523.681401] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1523.681771] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1523.682477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1523.685314] Lustre: Skipped 1 previous similar message [ 1528.801468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1528.803796] Lustre: Skipped 2 previous similar messages [ 1529.728964] LustreError: 95644:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1529.730942] LustreError: 95644:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1529.741764] Lustre: server umount lustre-MDT0000 complete [ 1531.214430] LustreError: 95845:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1531.217147] LustreError: 95845:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1531.312574] Lustre: server umount lustre-MDT0001 complete [ 1532.638019] Lustre: server umount MGS complete [ 1536.522174] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 01:20:46 (1759814446) [ 1545.979657] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 1549.178391] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 1552.580360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1554.767051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1556.833874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1558.772814] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1560.921131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1560.941510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1562.189051] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1565.784146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1565.808191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1565.882282] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1565.900539] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1565.937523] Lustre: lustre-MDT0001: new disk, initializing [ 1565.962493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1565.971652] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1565.974441] Lustre: 100553:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1565.978478] LustreError: 100553:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1567.154804] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1571.004176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1571.024271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1571.104822] Lustre: lustre-OST0000: new disk, initializing [ 1571.106346] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1571.126209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1572.823878] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1575.730468] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1576.929698] Lustre: 101910:0:(client.c:1627:after_reply()) @@@ resending request on EINPROGRESS req@0000000009cca9b1 x1845299143279104/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1759814503 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1577.787831] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1577.792965] Lustre: MGS: Client 0ddee5c7-5e8d-474d-9530-fbeaba981861 (at 0@lo) reconnecting [ 1577.806013] Lustre: 99838:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1577.813956] LustreError: 99838:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 1577.824987] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1577.857956] Lustre: lustre-MDT0000: new disk, initializing [ 1577.903072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1578.912817] Lustre: 102268:0:(client.c:1627:after_reply()) @@@ resending request on EINPROGRESS req@00000000a616cd6b x1845299143284224/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1759814505 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1578.977087] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1579.075936] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1586.018984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1586.021821] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1588.002051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 1588.004369] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 1588.068221] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 1608.673112] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1608.675026] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1608.678359] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1613.870116] LustreError: 102790:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 1613.872197] LustreError: 102790:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1613.933715] Lustre: server umount lustre-OST0000 complete [ 1615.329519] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1615.329669] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1615.334301] Lustre: Skipped 2 previous similar messages [ 1620.449852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1620.452502] Lustre: Skipped 5 previous similar messages [ 1621.438606] LustreError: 102991:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 1621.442053] LustreError: 102991:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1621.455067] Lustre: server umount lustre-MDT0000 complete [ 1628.874448] Lustre: server umount lustre-MDT0001 complete [ 1630.143966] LustreError: 103396:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1630.147238] LustreError: 103396:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1630.174508] Lustre: server umount MGS complete [ 1633.840207] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 01:22:23 (1759814543) [ 1639.811305] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 1640.918420] Key type lgssc unregistered [ 1641.051413] LNet: 104729:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1641.053727] LNetError: 104729:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1641.062329] LNet: Removed LNI 192.168.203.153@tcp [ 1641.380771] Key type .llcrypt unregistered [ 1641.381754] Key type ._llcrypt unregistered [ 1648.351061] Key type ._llcrypt registered [ 1648.352158] Key type .llcrypt registered [ 1648.393318] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 1648.830930] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1648.840300] alg: No test for adler32 (adler32-zlib) [ 1649.696495] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 1649.782683] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 1651.368155] Key type lgssc registered [ 1651.762167] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1654.417535] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1654.424221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1655.589331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1656.971616] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1659.966825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1660.104285] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1661.591514] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1663.544248] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1664.816603] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1667.172817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1667.289614] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1669.154154] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1670.942343] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1680.616058] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 1685.985774] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1685.986704] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1685.989917] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1685.995448] Lustre: Skipped 1 previous similar message [ 1686.010731] LustreError: 108174:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1686.066764] Lustre: server umount lustre-OST0000 complete [ 1689.472785] LustreError: 108375:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1689.475571] LustreError: 108375:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1689.501100] Lustre: server umount lustre-MDT0000 complete [ 1690.752714] LustreError: 105888:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759814600 with bad export cookie 4735855846698196932 [ 1690.756388] LustreError: 105888:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1690.756791] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1690.823034] LustreError: 108576:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1690.826058] LustreError: 108576:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1690.907820] Lustre: server umount lustre-MDT0001 complete [ 1694.690767] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 1695.814362] Key type lgssc unregistered [ 1695.955422] LNet: 109107:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1695.958384] LNetError: 109107:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1695.968363] LNet: Removed LNI 192.168.203.153@tcp [ 1696.307380] Key type .llcrypt unregistered [ 1696.308855] Key type ._llcrypt unregistered [ 1702.955033] Key type ._llcrypt registered [ 1702.956525] Key type .llcrypt registered [ 1702.999970] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 1703.370652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1703.435048] alg: No test for adler32 (adler32-zlib) [ 1704.296054] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 1704.380517] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 1705.968126] Key type lgssc registered [ 1706.344392] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1708.469291] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1708.471816] LustreError: 110147:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 1708.475129] LustreError: 110149:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 1708.477142] LustreError: 110147:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4997 [ 1708.479571] LustreError: 110147:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1708.483871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1708.484064] LustreError: 110147:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1709.650118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1712.213981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1712.307838] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1713.616867] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1715.240350] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing lsmod [ 1716.210928] LustreError: 111323:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 1716.211125] LustreError: 111324:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 1716.215897] LustreError: 111323:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4998 [ 1716.218137] LustreError: 111323:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1716.222114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1716.222256] LustreError: 111323:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1716.343906] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1717.292902] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1717.294896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1717.299906] Lustre: Skipped 2 previous similar messages [ 1717.301753] Lustre: Skipped 1 previous similar message [ 1722.339365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1722.341624] Lustre: Skipped 3 previous similar messages [ 1723.362608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1723.364970] Lustre: Skipped 1 previous similar message [ 1723.393197] LustreError: 111513:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1723.426373] Lustre: server umount lustre-MDT0000 complete [ 1724.681664] LustreError: 110164:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759814634 with bad export cookie 7000154197797410507 [ 1724.684540] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1724.686803] LustreError: 110164:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1725.408266] LustreError: 110620:0:(client.c:1370:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000000c94109b x1845299368706304/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1725.420139] LustreError: 111718:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1725.422637] LustreError: 111718:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1725.498677] Lustre: server umount lustre-MDT0001 complete [ 1736.690575] LustreError: 111918:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1736.693491] LustreError: 111918:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1736.762533] Lustre: server umount lustre-OST0000 complete [ 1739.482672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1739.640624] LustreError: 112395:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1739.646397] LustreError: 112395:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1739.662487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1740.923860] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1743.715237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1743.842493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1745.172605] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1747.017285] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1748.302126] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1750.556063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1752.610381] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1754.432540] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1764.008035] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 1767.011598] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1767.012874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1767.015678] Lustre: Skipped 1 previous similar message [ 1772.269195] LustreError: 114664:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1772.271256] LustreError: 114664:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1772.332231] Lustre: server umount lustre-OST0000 complete [ 1774.560853] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1774.563909] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1774.570413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1774.572588] Lustre: Skipped 1 previous similar message [ 1777.633913] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1779.710144] LustreError: 114865:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1779.712943] LustreError: 114865:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1779.744094] Lustre: server umount lustre-MDT0000 complete [ 1780.992612] LustreError: 112375:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759814691 with bad export cookie 7000154197797411193 [ 1780.996468] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1780.997432] LustreError: 112375:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1781.136401] Lustre: server umount lustre-MDT0001 complete [ 1784.863741] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 1785.902990] Key type lgssc unregistered [ 1786.031356] LNet: 115598:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1786.034132] LNetError: 115598:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1786.040267] LNet: Removed LNI 192.168.203.153@tcp [ 1786.348301] Key type .llcrypt unregistered [ 1786.349286] Key type ._llcrypt unregistered [ 1795.327074] Key type ._llcrypt registered [ 1795.328729] Key type .llcrypt registered [ 1795.367365] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 1795.709054] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1795.761036] alg: No test for adler32 (adler32-zlib) [ 1796.621708] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 1796.703892] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 1798.280207] Key type lgssc registered [ 1798.659995] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1801.207690] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1801.214492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1802.372904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1803.704405] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1806.508584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1806.625698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1807.976464] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1809.782262] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1811.070450] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1813.347333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1813.470188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1815.394401] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1817.276213] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1828.840191] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 1834.893939] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1835.672677] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1837.025025] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1837.027211] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1837.032140] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1839.073827] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1839.074869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1839.078143] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1842.733251] LustreError: 119343:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1842.791470] Lustre: server umount lustre-OST0000 complete [ 1844.193578] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1844.194122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1844.197279] Lustre: Skipped 1 previous similar message [ 1844.199223] Lustre: Skipped 1 previous similar message [ 1849.313027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1849.314838] Lustre: Skipped 2 previous similar messages [ 1850.172957] LustreError: 119548:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1850.176105] LustreError: 119548:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1850.203915] Lustre: server umount lustre-MDT0000 complete [ 1851.510444] LustreError: 116953:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759814761 with bad export cookie 16303012217748189665 [ 1851.514717] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1851.515053] LustreError: 116953:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1851.590626] LustreError: 119750:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1851.593945] LustreError: 119750:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1851.681222] Lustre: server umount lustre-MDT0001 complete [ 1855.576585] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 1856.784149] Key type lgssc unregistered [ 1856.939357] LNet: 120281:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1856.941649] LNetError: 120281:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1856.951321] LNet: Removed LNI 192.168.203.153@tcp [ 1857.291443] Key type .llcrypt unregistered [ 1857.292715] Key type ._llcrypt unregistered [ 1864.493827] Key type ._llcrypt registered [ 1864.495027] Key type .llcrypt registered [ 1864.532899] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 1864.968168] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1864.994120] alg: No test for adler32 (adler32-zlib) [ 1865.858460] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 1865.946048] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 1867.528164] Key type lgssc registered [ 1867.902397] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1870.620357] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1870.628067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1871.797974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1873.195589] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1875.920090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1876.049788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1877.328565] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1879.104272] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1880.336936] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1882.607390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1882.739521] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1884.683621] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1886.471282] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1897.064568] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 1907.682309] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1907.683395] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1907.684880] LustreError: Skipped 1 previous similar message [ 1907.685680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1907.687733] Lustre: Skipped 1 previous similar message [ 1907.693007] Lustre: Skipped 1 previous similar message [ 1910.827122] LustreError: 123725:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1910.885233] Lustre: server umount lustre-OST0000 complete [ 1912.801669] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1912.802433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1912.807569] Lustre: Skipped 1 previous similar message [ 1917.921166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1917.923155] Lustre: Skipped 2 previous similar messages [ 1918.331831] LustreError: 123929:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1918.333522] LustreError: 123929:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1918.358730] Lustre: server umount lustre-MDT0000 complete [ 1919.605070] LustreError: 123930:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759814829 with bad export cookie 15884616122981842507 [ 1919.607705] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1919.609157] LustreError: 123930:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1919.671239] LustreError: 124131:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1919.673721] LustreError: 124131:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1919.743328] Lustre: server umount lustre-MDT0001 complete [ 1923.282943] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 1924.318984] Key type lgssc unregistered [ 1924.445378] LNet: 124662:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1924.447556] LNetError: 124662:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1924.457251] LNet: Removed LNI 192.168.203.153@tcp [ 1924.747420] Key type .llcrypt unregistered [ 1924.748240] Key type ._llcrypt unregistered [ 1933.211651] Key type ._llcrypt registered [ 1933.212584] Key type .llcrypt registered [ 1933.246441] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 1933.530688] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1933.536935] alg: No test for adler32 (adler32-zlib) [ 1934.387245] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 1934.486674] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 1936.064100] Key type lgssc registered [ 1936.389630] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1938.757635] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1938.762977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1939.897786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1941.030463] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1943.585959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1943.696165] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1944.900503] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1946.540113] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1947.722386] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1949.743607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1949.843743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1950.887153] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 1951.535654] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 1953.251514] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1956.572394] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 1957.776825] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 1957.780600] Lustre: Skipped 2 previous similar messages [ 1959.131227] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 1959.134279] Lustre: Skipped 2 previous similar messages [ 1961.981881] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 1961.993158] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1965.297377] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.53@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 1965.300353] Lustre: Skipped 2 previous similar messages [ 1968.436203] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.53@tcp 192.168.0.[2,10]@tcp [ 1970.453354] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.53/24@tcp in log lustre-MDT0000 [ 1970.457751] Lustre: Skipped 2 previous similar messages [ 1976.311242] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.53/24@tcp [ 1976.313514] Lustre: Skipped 1 previous similar message [ 1978.849687] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1978.850277] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1978.854065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1978.857819] Lustre: Skipped 1 previous similar message [ 1983.970776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1983.972248] Lustre: Skipped 2 previous similar messages [ 1984.362416] LustreError: 129826:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1984.413970] Lustre: server umount lustre-OST0000 complete [ 1989.089736] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1989.090255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1989.092837] Lustre: Skipped 1 previous similar message [ 1989.097080] Lustre: Skipped 1 previous similar message [ 1991.677146] LustreError: 130027:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1991.679872] LustreError: 130027:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1991.709534] Lustre: server umount lustre-MDT0000 complete [ 1992.931064] LustreError: 126015:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759814903 with bad export cookie 8560708769124726527 [ 1992.933894] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1992.934976] LustreError: 126015:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1992.993029] LustreError: 130228:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1992.995157] LustreError: 130228:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1993.073784] Lustre: server umount lustre-MDT0001 complete [ 1996.443217] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 1997.438537] Key type lgssc unregistered [ 1997.572363] LNet: 130760:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1997.575147] LNetError: 130760:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1997.586324] LNet: Removed LNI 192.168.203.153@tcp [ 1997.850638] Key type .llcrypt unregistered [ 1997.852114] Key type ._llcrypt unregistered [ 2007.586098] Key type ._llcrypt registered [ 2007.587445] Key type .llcrypt registered [ 2007.624795] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 2007.948643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2007.999063] alg: No test for adler32 (adler32-zlib) [ 2008.846301] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 2008.922135] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 2010.504170] Key type lgssc registered [ 2010.870846] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2013.380618] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2013.386495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2014.523845] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2014.529795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2014.532364] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.53/24@tcp [ 2015.768379] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2018.438170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2018.535992] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2018.543250] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.53/24@tcp [ 2018.567109] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2019.849400] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2021.650935] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2022.843511] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2025.090392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2025.203166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2027.054433] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2028.728869] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2040.551462] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 2048.992815] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2048.994819] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2048.998209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2051.043731] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2051.044364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2051.047266] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2051.049802] Lustre: Skipped 1 previous similar message [ 2053.356728] LustreError: 134601:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2053.427539] Lustre: server umount lustre-OST0000 complete [ 2056.162164] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2056.162729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2056.168597] Lustre: Skipped 1 previous similar message [ 2060.797242] LustreError: 134803:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2060.798919] LustreError: 134803:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2060.826939] Lustre: server umount lustre-MDT0000 complete [ 2061.281135] LustreError: 133007:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2061.285489] LustreError: 133007:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2062.073236] LustreError: 134515:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759814972 with bad export cookie 5894124686126016604 [ 2062.076820] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2062.078059] LustreError: 134515:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2062.145819] LustreError: 135005:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2062.147471] LustreError: 135005:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2062.229269] Lustre: server umount lustre-MDT0001 complete [ 2065.933879] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 2067.009601] Key type lgssc unregistered [ 2067.157470] LNet: 135536:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2067.159602] LNetError: 135536:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2067.166361] LNet: Removed LNI 192.168.203.153@tcp [ 2067.470621] Key type .llcrypt unregistered [ 2067.471665] Key type ._llcrypt unregistered [ 2075.741370] Key type ._llcrypt registered [ 2075.742341] Key type .llcrypt registered [ 2075.784813] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 2082.216125] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 2082.712409] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2082.725887] alg: No test for adler32 (adler32-zlib) [ 2083.583728] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 2083.680087] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 2085.272276] Key type lgssc registered [ 2085.713313] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2088.806915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2091.008567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2093.100240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2095.243758] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2097.418417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2097.439626] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2097.445495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2098.533720] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2098.545265] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2098.595615] Lustre: lustre-MDT0000: new disk, initializing [ 2098.622530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2098.631393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2099.953874] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2104.128930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2104.150806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2104.182940] Lustre: 138881:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2104.208592] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2104.210263] Lustre: Skipped 1 previous similar message [ 2104.252311] Lustre: lustre-MDT0001: new disk, initializing [ 2104.295134] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2104.309351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2104.314230] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2105.682080] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2108.854697] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2110.173197] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2112.621484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2112.649189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2112.759406] Lustre: lustre-OST0000: new disk, initializing [ 2112.762310] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2112.793027] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2114.742463] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2117.881850] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2119.254172] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2122.234071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2122.237712] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2122.269974] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2122.401318] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 2123.806194] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2123.879182] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2127.330236] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2127.332548] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2127.333412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2127.336167] Lustre: Skipped 1 previous similar message [ 2130.608148] LustreError: 141841:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2130.694408] Lustre: server umount lustre-OST0000 complete [ 2132.449451] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2132.450071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2132.454464] Lustre: Skipped 1 previous similar message [ 2132.457143] Lustre: Skipped 1 previous similar message [ 2137.569794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2137.572019] Lustre: Skipped 1 previous similar message [ 2138.175548] LustreError: 142042:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2138.177696] LustreError: 142042:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2138.205137] Lustre: server umount lustre-MDT0000 complete [ 2139.457438] LustreError: 138873:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759815049 with bad export cookie 18375666731593403459 [ 2139.460993] LustreError: 138873:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2139.461185] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2139.527452] LustreError: 142245:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2139.529803] LustreError: 142245:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2139.638425] Lustre: server umount lustre-MDT0001 complete [ 2142.472101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2142.663106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2143.980605] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2146.958497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2147.096818] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2148.497302] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2150.254915] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2151.489204] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2153.904477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2156.018926] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2157.915665] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2159.271873] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2167.513913] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 2168.845904] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2168.924082] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2172.459459] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2172.487893] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2172.552222] Lustre: lustre-OST0001: new disk, initializing [ 2172.555861] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2172.586542] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2172.590132] Lustre: Skipped 1 previous similar message [ 2174.587355] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2177.933169] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2178.052289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2178.055616] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2178.087447] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2179.038363] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 2180.428948] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2180.506821] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2181.789456] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff89d786c3c800.ost_server_uuid 50 [ 2182.332552] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff89d786c3c800.ost_server_uuid in FULL state after 0 sec [ 2188.260326] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2188.263194] LustreError: Skipped 1 previous similar message [ 2188.264383] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2188.268577] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2188.270693] Lustre: Skipped 1 previous similar message [ 2190.382080] LustreError: 147029:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 2190.384697] LustreError: 147029:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2190.404764] Lustre: server umount lustre-OST0001 complete [ 2193.378466] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2193.379661] LustreError: 144318:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2193.380016] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2193.380022] Lustre: Skipped 1 previous similar message [ 2193.381159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2193.381425] LustreError: Skipped 2 previous similar messages [ 2193.385758] LustreError: 144318:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2197.742131] LustreError: 147230:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2197.744867] LustreError: 147230:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2197.807181] Lustre: server umount lustre-OST0000 complete [ 2203.619092] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2203.619372] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2203.620504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2203.620512] Lustre: Skipped 1 previous similar message [ 2203.627925] Lustre: Skipped 2 previous similar messages [ 2205.278086] Lustre: server umount lustre-MDT0000 complete [ 2206.529043] LustreError: 145778:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759815116 with bad export cookie 18375666731593404313 [ 2206.532166] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2206.534033] LustreError: 145778:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2206.603305] LustreError: 147633:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 2206.605742] LustreError: 147633:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2206.685772] Lustre: server umount lustre-MDT0001 complete [ 2213.394523] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 2214.471468] Key type lgssc unregistered [ 2214.605329] LNet: 148621:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2214.608701] LNetError: 148621:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2214.618292] LNet: Removed LNI 192.168.203.153@tcp [ 2214.960525] Key type .llcrypt unregistered [ 2214.961607] Key type ._llcrypt unregistered [ 2225.762232] Key type ._llcrypt registered [ 2225.763811] Key type .llcrypt registered [ 2225.799857] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 2232.237074] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 2232.569923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2232.576498] alg: No test for adler32 (adler32-zlib) [ 2233.427168] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 2233.516719] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 2235.096137] Key type lgssc registered [ 2235.482992] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2238.492983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2240.753960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2242.739603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2244.843664] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2247.077809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2247.099618] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2247.105641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2248.188170] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2248.197903] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2248.234122] Lustre: lustre-MDT0000: new disk, initializing [ 2248.257621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2248.265249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2249.464640] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2253.454728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2253.477319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2253.505854] Lustre: 152544:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2253.529852] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2253.532160] Lustre: Skipped 1 previous similar message [ 2253.572778] Lustre: lustre-MDT0001: new disk, initializing [ 2253.603452] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2253.616336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2253.620814] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2254.825815] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2257.838205] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2258.960205] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2261.145321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2261.167799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2261.264340] Lustre: lustre-OST0000: new disk, initializing [ 2261.266919] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2261.297654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2263.209585] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2266.362827] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2266.622161] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2266.627821] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2266.657695] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2267.692079] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2267.761492] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2269.116881] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2269.193502] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2271.714781] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2271.716419] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2271.717746] LustreError: Skipped 1 previous similar message [ 2271.718455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2271.723137] Lustre: Skipped 1 previous similar message [ 2271.856130] LustreError: 155479:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2271.921789] Lustre: server umount lustre-OST0000 complete [ 2274.272917] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2274.275876] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2274.280859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2274.282530] Lustre: Skipped 1 previous similar message [ 2276.833474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2276.834283] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2279.290095] LustreError: 155681:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2279.292416] LustreError: 155681:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2279.316328] Lustre: server umount lustre-MDT0000 complete [ 2280.511160] LustreError: 154288:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759815190 with bad export cookie 5793557825919776054 [ 2280.514190] LustreError: 154288:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2280.514411] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2280.571737] LustreError: 155884:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2280.573379] LustreError: 155884:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2280.651601] Lustre: server umount lustre-MDT0001 complete [ 2283.401276] Lustre: DEBUG MARKER: == conf-sanity test 47a: server restart does not lose lru_resize ========================================================== 01:33:13 (1759815193) [ 2288.426509] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 2291.395227] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 2294.611768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2296.826733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2298.899795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2301.081211] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2303.355878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2303.374563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2303.456567] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2303.467981] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2303.506302] Lustre: lustre-MDT0000: new disk, initializing [ 2303.532031] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2303.540194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2304.902455] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2309.156971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2309.180511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2309.218580] Lustre: 158875:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2309.242760] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2309.246154] Lustre: Skipped 1 previous similar message [ 2309.283679] Lustre: lustre-MDT0001: new disk, initializing [ 2309.314738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2309.325994] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2309.329604] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2310.637409] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2313.737227] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2314.957065] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2317.222461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2317.242525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2317.338454] Lustre: lustre-OST0000: new disk, initializing [ 2317.340182] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2317.369768] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2319.226226] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2322.235854] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2326.521479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2326.525880] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2326.553808] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2329.349220] Lustre: Failing over lustre-OST0000 [ 2329.387119] LustreError: 161317:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2329.390276] LustreError: 161317:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2329.451609] Lustre: server umount lustre-OST0000 complete [ 2342.339483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2342.453981] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2342.456716] Lustre: Skipped 1 previous similar message [ 2342.463476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2342.470169] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2343.937347] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2344.320817] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2345.560544] Lustre: Failing over lustre-MDT0000 [ 2345.624182] LustreError: 162289:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2345.626397] LustreError: 162289:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2345.651377] Lustre: server umount lustre-MDT0000 complete [ 2347.488280] Lustre: 150758:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759815241/real 1759815241] req@00000000287dabca x1845299923442944/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1759815257 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2347.489899] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2347.501592] Lustre: Skipped 1 previous similar message [ 2352.608128] Lustre: 150759:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759815246/real 1759815246] req@0000000059d79924 x1845299923444992/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1759815262 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2352.609639] LustreError: 158885:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2352.622954] LustreError: 158885:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2354.167403] LustreError: 158882:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2357.664086] Lustre: 150760:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759815251/real 1759815251] req@000000009a18a540 x1845299923446144/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1759815267 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2357.729141] LustreError: 158885:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2357.737116] LustreError: 158885:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2358.415369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2358.451962] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2358.456572] Lustre: 158877:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.203.153@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2358.528160] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2358.575443] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2359.287690] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2359.832261] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2363.878368] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2363.884542] Lustre: lustre-OST0000: Recovery over after 0:20, of 3 clients 3 recovered and 0 were evicted. [ 2363.886904] Lustre: Skipped 1 previous similar message [ 2363.909826] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2368.996886] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2368.996924] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2369.000135] LustreError: Skipped 1 previous similar message [ 2369.000681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2369.004191] Lustre: Skipped 2 previous similar messages [ 2369.005472] Lustre: Skipped 1 previous similar message [ 2370.861180] LustreError: 163177:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2370.862903] LustreError: 163177:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2370.919503] Lustre: server umount lustre-OST0000 complete [ 2374.114119] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2374.114754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2374.118774] Lustre: Skipped 2 previous similar messages [ 2384.353762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2384.356731] Lustre: Skipped 3 previous similar messages [ 2386.400185] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2386.464574] Lustre: server umount lustre-MDT0000 complete [ 2387.648657] LustreError: 159656:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759815297 with bad export cookie 5793557825919778112 [ 2387.652261] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2387.654196] LustreError: 159656:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2387.715313] LustreError: 163581:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2387.717199] LustreError: 163581:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 2387.792895] Lustre: server umount lustre-MDT0001 complete [ 2391.322177] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 2392.395582] Key type lgssc unregistered [ 2392.532364] LNet: 164111:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2392.534882] LNetError: 164111:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2392.540336] LNet: Removed LNI 192.168.203.153@tcp [ 2392.837264] Key type .llcrypt unregistered [ 2392.838299] Key type ._llcrypt unregistered [ 2413.039154] Key type ._llcrypt registered [ 2413.040076] Key type .llcrypt registered [ 2413.076937] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 2413.405709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2413.412649] alg: No test for adler32 (adler32-zlib) [ 2414.261579] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 2414.342797] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 2415.920134] Key type lgssc registered [ 2416.274459] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2418.869106] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2418.877171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2420.041213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2421.306466] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2424.334291] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2426.357095] LustreError: 165637:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2426.361967] LustreError: 165637:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2426.364502] LustreError: 165637:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -5 [ 2426.425862] LustreError: 166160:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2426.522356] Lustre: server umount lustre-MDT0000 complete [ 2431.296836] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 2434.440983] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 2437.765856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2439.980956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2441.998339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2444.152659] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2446.376050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2446.398404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2446.483873] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2446.495837] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2446.533614] Lustre: lustre-MDT0000: new disk, initializing [ 2446.558119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2446.564451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2447.776463] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2451.701804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2451.726995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2451.760504] Lustre: 168800:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2451.782590] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2451.784434] Lustre: Skipped 1 previous similar message [ 2451.819927] Lustre: lustre-MDT0001: new disk, initializing [ 2451.850256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2451.861425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2451.864468] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2453.103485] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2456.053216] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2457.241352] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2459.390308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2459.415664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2459.513554] Lustre: lustre-OST0000: new disk, initializing [ 2459.515560] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2459.545204] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2461.240714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2461.243496] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2461.281193] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2461.369571] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2464.442903] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2466.244679] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2466.837843] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 01:36:16 (1759815376) [ 2528.736478] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2528.739525] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2528.743512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2532.832560] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2532.832714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2532.835958] Lustre: Skipped 1 previous similar message [ 2532.837804] Lustre: Skipped 1 previous similar message [ 2534.656991] LustreError: 172299:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2534.659281] LustreError: 172299:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2534.699465] Lustre: server umount lustre-MDT0000 complete [ 2535.876390] LustreError: 172200:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759815445 with bad export cookie 2885869028626347302 [ 2535.877461] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2535.879170] LustreError: 172200:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2537.952363] LustreError: 171152:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2537.952506] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2537.956198] LustreError: 171152:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2537.961068] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2537.962678] Lustre: Skipped 1 previous similar message [ 2540.960506] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2540.960506] LustreError: 168807:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2545.056541] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2545.056543] LustreError: 171152:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2550.240165] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2550.240560] LustreError: 171152:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2550.276439] LustreError: 172502:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2550.280420] LustreError: 172502:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2550.354255] Lustre: server umount lustre-MDT0001 complete [ 2557.673501] LustreError: 172704:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2557.676728] LustreError: 172704:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2557.733688] Lustre: server umount lustre-OST0000 complete [ 2560.719308] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 2564.318321] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 2567.776892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2569.968754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2572.006673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2574.124717] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2576.317355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2576.340953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2576.427199] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2576.438069] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2576.478227] Lustre: lustre-MDT0000: new disk, initializing [ 2576.505760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2576.512218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2577.800263] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2582.054760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2582.080242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2582.115256] Lustre: 175039:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2582.143446] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2582.145061] Lustre: Skipped 1 previous similar message [ 2582.186750] Lustre: lustre-MDT0001: new disk, initializing [ 2582.231431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2582.235480] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2583.568549] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2586.654924] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2587.826882] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2590.064802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2590.088483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2590.197500] Lustre: lustre-OST0000: new disk, initializing [ 2590.199382] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2590.229112] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2590.231454] Lustre: Skipped 1 previous similar message [ 2591.030512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2591.033041] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2591.080178] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2592.155209] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2595.194221] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2596.628768] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2596.711367] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2598.045728] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2598.137944] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2600.814082] LustreError: 177974:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 2600.816150] LustreError: 177974:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2600.888942] Lustre: server umount lustre-OST0000 complete [ 2602.976967] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2602.980657] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2602.987097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2602.989092] Lustre: Skipped 1 previous similar message [ 2612.708225] LustreError: 178174:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2612.710274] LustreError: 178174:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2612.738673] Lustre: server umount lustre-MDT0000 complete [ 2614.014106] LustreError: 178176:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759815524 with bad export cookie 2885869028626506118 [ 2614.017482] LustreError: 178176:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2614.017878] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2624.577553] Lustre: server umount lustre-MDT0001 complete [ 2627.630072] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 01:38:57 (1759815537) [ 2633.046753] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 2636.140281] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 2639.796197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2641.981300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2643.895288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2645.937724] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2648.163142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2648.184943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2648.267693] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2648.278473] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2648.320357] Lustre: lustre-MDT0000: new disk, initializing [ 2648.347462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2648.354028] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2649.683304] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2653.772646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2653.793613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2653.827345] Lustre: 181363:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2653.923062] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2655.177833] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2658.217685] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2659.418230] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2661.622179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2661.648332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2663.603723] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2666.571963] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2668.535601] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2668.566803] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2678.755561] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2678.756020] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2678.758521] LustreError: Skipped 1 previous similar message [ 2678.762120] Lustre: Skipped 1 previous similar message [ 2678.764722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2678.766705] Lustre: Skipped 5 previous similar messages [ 2679.982985] LustreError: 183910:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2679.985130] LustreError: 183910:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2680.057241] Lustre: server umount lustre-OST0000 complete [ 2687.450073] Lustre: server umount lustre-MDT0000 complete [ 2688.700310] LustreError: 181355:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759815598 with bad export cookie 2885869028626506930 [ 2688.703209] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2688.706128] LustreError: 181355:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2688.850824] Lustre: server umount lustre-MDT0001 complete [ 2691.536776] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 01:40:01 (1759815601) [ 2696.635739] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 2699.648590] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 2703.115787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2705.185360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2707.128412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2709.194051] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2711.512312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2711.534131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2711.619062] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2711.632127] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2711.633837] Lustre: Skipped 3 previous similar messages [ 2711.678611] Lustre: lustre-MDT0000: new disk, initializing [ 2711.679981] Lustre: Skipped 2 previous similar messages [ 2711.707274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2711.709237] Lustre: Skipped 2 previous similar messages [ 2711.714689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2711.717700] Lustre: Skipped 2 previous similar messages [ 2713.038368] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2717.228944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2717.254536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2717.289335] Lustre: 187306:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2717.399148] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2718.761351] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2721.949888] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2723.109162] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2725.341498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2725.363286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2727.404886] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2728.235060] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2730.514732] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2733.541483] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2733.541625] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2733.543461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2733.543468] Lustre: Skipped 2 previous similar messages [ 2733.545040] LustreError: Skipped 1 previous similar message [ 2733.554472] Lustre: Skipped 3 previous similar messages [ 2738.541103] LustreError: 189856:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2738.544321] LustreError: 189856:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2738.613696] Lustre: server umount lustre-OST0000 complete [ 2743.265000] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2746.015609] Lustre: server umount lustre-MDT0000 complete [ 2747.285355] LustreError: 189064:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759815657 with bad export cookie 2885869028626508001 [ 2747.287943] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2747.288775] LustreError: 189064:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2747.446662] Lustre: server umount lustre-MDT0001 complete [ 2751.125315] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 2752.249379] Key type lgssc unregistered [ 2752.379387] LNet: 190789:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2752.381776] LNetError: 190789:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2752.392303] LNet: Removed LNI 192.168.203.153@tcp [ 2752.724814] Key type .llcrypt unregistered [ 2752.726025] Key type ._llcrypt unregistered [ 2761.547078] Key type ._llcrypt registered [ 2761.548701] Key type .llcrypt registered [ 2761.596113] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 2761.991991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2762.002834] alg: No test for adler32 (adler32-zlib) [ 2762.888974] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 2762.999699] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 2764.608158] Key type lgssc registered [ 2765.060132] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2767.832386] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2767.840504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2769.001995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2770.242144] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2773.081895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2773.248248] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2774.748809] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2776.758250] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2778.107755] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2780.386516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2780.498969] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2782.371737] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2784.153609] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2794.855626] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2814.433543] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2814.436494] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2814.440357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2815.460239] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2815.460663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2815.460994] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2815.467962] Lustre: Skipped 1 previous similar message [ 2817.265131] LustreError: 194436:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2817.356011] Lustre: server umount lustre-OST0000 complete [ 2819.553024] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2819.554903] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2819.560494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2824.707836] LustreError: 194638:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2824.709699] LustreError: 194638:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2824.739231] Lustre: server umount lustre-MDT0000 complete [ 2825.697271] LustreError: 194331:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2825.702901] LustreError: 194331:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2826.010521] LustreError: 192146:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759815736 with bad export cookie 2199412025036654127 [ 2826.013169] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2826.013689] LustreError: 192146:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2826.076994] LustreError: 194841:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2826.078848] LustreError: 194841:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2826.156517] Lustre: server umount lustre-MDT0001 complete [ 2829.662726] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 2830.617882] Key type lgssc unregistered [ 2830.744344] LNet: 195371:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2830.746668] LNetError: 195371:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2830.756323] LNet: Removed LNI 192.168.203.153@tcp [ 2831.070182] Key type .llcrypt unregistered [ 2831.071132] Key type ._llcrypt unregistered [ 2839.667728] Key type ._llcrypt registered [ 2839.668629] Key type .llcrypt registered [ 2839.701447] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 2840.107624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2840.115756] alg: No test for adler32 (adler32-zlib) [ 2840.969762] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 2841.053229] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 2842.632126] Key type lgssc registered [ 2842.981861] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2845.425313] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2845.431431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2846.582309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2847.872542] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2850.690321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2850.822676] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2852.125545] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2853.921839] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2855.147836] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2857.347855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2857.461140] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2859.338677] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2861.137529] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2872.807818] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 2880.992729] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2880.994583] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2880.997813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2882.045639] Lustre: lustre-OST0000: Not available for connect from 192.168.203.53@tcp (stopping) [ 2883.043223] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2883.048250] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2883.048458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2883.053546] Lustre: Skipped 1 previous similar message [ 2887.160198] Lustre: lustre-OST0000: Not available for connect from 192.168.203.53@tcp (stopping) [ 2892.280374] Lustre: lustre-OST0000: Not available for connect from 192.168.203.53@tcp (stopping) [ 2892.283528] Lustre: Skipped 2 previous similar messages [ 2893.280143] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2893.302284] LustreError: 199018:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2893.360342] Lustre: server umount lustre-OST0000 complete [ 2895.090065] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff89d786c3a000.ost_server_uuid 50 [ 2895.579523] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff89d786c3a000.ost_server_uuid in DISCONN state after 0 sec [ 2896.080158] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 2906.195815] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2906.196400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2906.200169] Lustre: Skipped 1 previous similar message [ 2906.202430] Lustre: Skipped 2 previous similar messages [ 2912.319783] LustreError: 199415:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2912.321968] LustreError: 199415:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2912.345987] Lustre: server umount lustre-MDT0000 complete [ 2913.544211] LustreError: 196728:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759815823 with bad export cookie 10747530518204884993 [ 2913.547445] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2913.547515] LustreError: 196728:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2913.616233] LustreError: 199616:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2913.618436] LustreError: 199616:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2913.702329] Lustre: server umount lustre-MDT0001 complete [ 2916.368899] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 01:43:46 (1759815826) [ 2918.498155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2918.641375] LustreError: 200285:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2918.646754] LustreError: 200285:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2918.665692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2919.875775] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2922.596267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2922.710139] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2923.925989] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2925.630324] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2926.813146] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2928.987076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2930.154282] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 2930.942161] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2932.717091] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2935.012829] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2935.036647] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2935.100865] Lustre: lustre-OST0001: new disk, initializing [ 2935.102943] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2935.139931] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2935.141636] Lustre: Skipped 1 previous similar message [ 2936.956202] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2940.051192] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2940.932347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2940.938025] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2940.979514] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2942.432795] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2942.435278] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2942.439098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2942.440833] Lustre: Skipped 2 previous similar messages [ 2945.504844] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2955.232130] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2955.250122] LustreError: 203489:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2955.253361] LustreError: 203489:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2955.269914] Lustre: server umount lustre-OST0000 complete [ 2955.769161] LustreError: 201881:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2957.170186] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2957.241480] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2958.542582] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2958.612312] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 2960.887860] LustreError: 201881:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2960.892568] LustreError: 201881:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2966.008908] LustreError: 203491:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2966.015579] LustreError: 203491:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2971.618666] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2971.620439] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2971.620510] LustreError: 201880:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2971.620524] LustreError: 201880:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2971.620753] LustreError: Skipped 1 previous similar message [ 2971.625358] Lustre: Skipped 1 previous similar message [ 2974.636088] LustreError: 204193:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 2974.638115] LustreError: 204193:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2974.716936] Lustre: server umount lustre-OST0001 complete [ 2976.739854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2976.741638] Lustre: Skipped 10 previous similar messages [ 2982.168404] Lustre: server umount lustre-MDT0000 complete [ 2983.375202] LustreError: 200267:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759815893 with bad export cookie 10747530518204886050 [ 2983.378906] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2983.378990] LustreError: 200267:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2983.452063] LustreError: 204597:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 2983.454577] LustreError: 204597:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2983.549808] Lustre: server umount lustre-MDT0001 complete [ 2985.426204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2986.675424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2987.913363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2989.179462] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2991.392816] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 01:45:01 (1759815901) [ 2993.554289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2993.644499] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2993.656482] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2993.734974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2995.123594] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 2997.960079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2997.991838] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2998.005621] Lustre: 206087:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2999.445579] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3001.217495] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3002.395099] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3004.643706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3004.711455] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3006.648590] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3008.497774] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3010.836707] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3010.862030] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3012.881953] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3013.100779] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3014.771325] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3018.213702] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3018.214147] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3018.217182] Lustre: Skipped 3 previous similar messages [ 3018.220597] LustreError: Skipped 1 previous similar message [ 3021.743617] LustreError: 209277:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3021.745298] LustreError: 209277:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 3021.764136] Lustre: server umount lustre-OST0000 complete [ 3023.333611] LustreError: 208997:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3023.337704] LustreError: 208997:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3038.506901] Lustre: server umount lustre-OST0001 complete [ 3043.809529] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3043.810320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3043.815321] Lustre: Skipped 3 previous similar messages [ 3043.817093] Lustre: Skipped 7 previous similar messages [ 3045.983699] Lustre: server umount lustre-MDT0000 complete [ 3047.300387] LustreError: 206079:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759815957 with bad export cookie 10747530518204887443 [ 3047.304214] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3047.306115] LustreError: 206079:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3047.456352] Lustre: server umount lustre-MDT0001 complete [ 3049.498047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3050.947700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3052.227943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3053.524266] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3055.712741] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 01:46:05 (1759815965) [ 3060.892682] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 3063.843830] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 3067.041050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3069.123421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3071.039228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3073.125638] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3075.533276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3075.559826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3075.645652] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3075.659174] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3075.703424] Lustre: lustre-MDT0000: new disk, initializing [ 3075.727318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3075.729257] Lustre: Skipped 3 previous similar messages [ 3075.734319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3077.134864] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3081.338320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3081.363196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3081.400328] Lustre: 213698:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3081.428702] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3081.430202] Lustre: Skipped 1 previous similar message [ 3081.466293] Lustre: lustre-MDT0001: new disk, initializing [ 3081.511973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3081.516384] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3082.856582] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3085.993507] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3087.319043] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3089.547259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3089.575338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3089.682037] Lustre: lustre-OST0000: new disk, initializing [ 3089.684733] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3091.651247] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3093.367347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3093.370221] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3093.400145] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3094.715250] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3096.091912] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3096.167344] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3097.528433] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3097.607040] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3098.596446] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3098.596931] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3098.598931] LustreError: Skipped 3 previous similar messages [ 3098.601785] Lustre: Skipped 1 previous similar message [ 3100.335124] LustreError: 216633:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3100.337919] LustreError: 216633:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 3100.404360] Lustre: server umount lustre-OST0000 complete [ 3107.738524] Lustre: server umount lustre-MDT0000 complete [ 3108.833522] LustreError: 214492:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3108.838489] LustreError: 214492:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 3109.024384] LustreError: 213689:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759816019 with bad export cookie 10747530518204888892 [ 3109.027719] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3109.029850] LustreError: 213689:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3109.185148] Lustre: server umount lustre-MDT0001 complete [ 3112.078572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3113.706712] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3116.541969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3118.073314] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3119.986355] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3121.215494] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3123.558412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3125.568665] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3127.388627] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3128.818561] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3138.094473] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 3139.439809] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3139.510292] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3142.113177] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3142.116369] LustreError: Skipped 1 previous similar message [ 3146.286312] Lustre: server umount lustre-OST0000 complete [ 3148.149182] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3148.224794] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3149.494226] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3173.050045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3173.171730] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3173.174531] Lustre: Skipped 5 previous similar messages [ 3175.004601] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3176.756971] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3177.298017] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3183.393529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3183.395056] Lustre: Skipped 9 previous similar messages [ 3184.299147] LustreError: 221970:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3184.301427] LustreError: 221970:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 3184.357219] Lustre: server umount lustre-OST0000 complete [ 3188.192867] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3188.197829] Lustre: Skipped 4 previous similar messages [ 3191.763339] Lustre: server umount lustre-MDT0000 complete [ 3192.974104] LustreError: 217495:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759816103 with bad export cookie 10747530518204889760 [ 3192.978695] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3192.980067] LustreError: 217495:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3193.124606] Lustre: server umount lustre-MDT0001 complete [ 3195.906395] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 01:48:26 (1759816106) [ 3198.159404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3198.309462] LustreError: 223048:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3198.315914] LustreError: 223048:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3199.583371] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3202.404336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3203.831782] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3205.664659] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3206.886892] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3209.076145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3211.057763] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3212.841452] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3214.213518] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3224.505756] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec [ 3225.849862] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3225.926805] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3228.182605] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3228.211843] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3228.273252] Lustre: lustre-OST0001: new disk, initializing [ 3228.275913] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3230.257202] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3233.378301] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3234.835779] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3236.355615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3236.358506] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3236.393120] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3236.961766] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 3238.371244] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3238.448369] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3241.445204] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3241.447275] LustreError: Skipped 3 previous similar messages [ 3245.184269] Lustre: server umount lustre-OST0001 complete [ 3247.169317] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3247.247173] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3248.619499] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3248.699192] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3264.504280] LustreError: 224642:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3264.509444] LustreError: 224642:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 3272.301570] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3274.324994] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3276.247893] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3276.848700] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3283.577657] Lustre: server umount lustre-OST0001 complete [ 3301.611103] Lustre: server umount lustre-OST0000 complete [ 3309.207200] Lustre: server umount lustre-MDT0000 complete [ 3310.421115] LustreError: 223027:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759816220 with bad export cookie 10747530518204890691 [ 3310.424022] LustreError: 223027:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3310.424276] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3316.541673] LustreError: 229613:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 3316.543476] LustreError: 229613:0:(obd_class.h:479:obd_check_dev()) Skipped 37 previous similar messages [ 3316.626619] Lustre: server umount lustre-MDT0001 complete [ 3318.537240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3319.854741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3321.135742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3322.408294] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3324.600653] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 01:50:34 (1759816234) [ 3326.868559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3326.969569] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3326.979767] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3327.051894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3327.053880] Lustre: Skipped 5 previous similar messages [ 3328.401576] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3331.228447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3331.263408] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3331.274330] Lustre: 231104:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3332.671705] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3334.435074] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3335.584205] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3337.876614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3337.946495] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3339.959636] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3341.831768] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3351.168695] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3351.191691] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3353.198093] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3354.942697] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3359.549774] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 4 sec [ 3360.860370] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3360.932966] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3362.164335] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3362.236381] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3363.361970] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff89d788543000.ost_server_uuid 50 [ 3363.825653] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff89d788543000.ost_server_uuid in FULL state after 0 sec [ 3364.142944] Lustre: Permanently deactivating lustre-OST0001 [ 3364.147322] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3364.947709] Lustre: Permanently reactivating lustre-OST0001 [ 3369.958149] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3369.960252] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3369.964241] Lustre: Skipped 7 previous similar messages [ 3371.968462] Lustre: server umount lustre-OST0001 complete [ 3379.373323] Lustre: server umount lustre-OST0000 complete [ 3386.715279] Lustre: server umount lustre-MDT0000 complete [ 3387.914266] LustreError: 231096:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759816298 with bad export cookie 10747530518204891951 [ 3387.917500] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3387.917504] LustreError: 231096:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3388.067192] Lustre: server umount lustre-MDT0001 complete [ 3389.914054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3391.180671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3392.412387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3393.676445] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3395.860971] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 01:51:45 (1759816305) [ 3396.182104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3398.375888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3398.473032] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3398.487289] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3398.489901] Lustre: Skipped 5 previous similar messages [ 3399.795966] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3402.575730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3402.604412] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3402.617296] Lustre: 237240:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3403.992018] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3405.722445] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3406.857953] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3408.977252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3408.999194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3409.067417] Lustre: Permanently deactivating lustre-OST0000 [ 3409.072540] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3410.930734] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3412.659111] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3413.105698] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3414.928939] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3416.969243] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3418.799092] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3419.676578] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3424.651798] Lustre: Permanently reactivating lustre-OST0000 [ 3426.613882] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3426.634292] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3435.711193] Lustre: server umount lustre-OST0001 complete [ 3440.099878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3440.100348] LustreError: 240406:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3440.101371] Lustre: Skipped 31 previous similar messages [ 3440.107583] LustreError: 240406:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 3451.360123] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3451.431771] Lustre: server umount lustre-OST0000 complete [ 3458.921580] Lustre: server umount lustre-MDT0000 complete [ 3460.138466] LustreError: 237971:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759816370 with bad export cookie 10747530518204893589 [ 3460.141213] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3460.141827] LustreError: 237971:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3460.286263] Lustre: server umount lustre-MDT0001 complete [ 3463.796130] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 3464.870122] Key type lgssc unregistered [ 3465.001268] LNet: 241636:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3465.004635] LNetError: 241636:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3465.012258] LNet: Removed LNI 192.168.203.153@tcp [ 3465.342041] Key type .llcrypt unregistered [ 3465.343188] Key type ._llcrypt unregistered [ 3473.610192] Key type ._llcrypt registered [ 3473.611044] Key type .llcrypt registered [ 3473.660893] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 3474.013172] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3474.042131] alg: No test for adler32 (adler32-zlib) [ 3474.893470] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 3474.984776] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 3476.568164] Key type lgssc registered [ 3476.948343] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3478.111720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3480.376359] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3480.382549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3481.536535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3482.882444] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3485.611694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3485.637053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3485.665949] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3485.675779] Lustre: 243052:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3485.679291] Lustre: Permanently deactivating lustre-MDT0001 [ 3485.683925] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3485.805377] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3487.065534] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3487.407417] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3487.409125] LustreError: 243092:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3487.411765] LustreError: 243092:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3487.414153] LustreError: 243092:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 1, failed: rc = -108 [ 3488.757445] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3489.868676] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3492.029198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3492.148796] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3493.840373] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3495.205002] LustreError: 244677:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3495.207839] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3495.209957] LustreError: 244677:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3495.603850] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3497.819775] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3497.886379] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3498.545955] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3499.776720] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3501.539263] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3506.729417] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3506.729535] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3512.377537] Lustre: Permanently reactivating lustre-MDT0001 [ 3512.383584] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3512.385177] Lustre: Skipped 2 previous similar messages [ 3519.997295] Lustre: lustre-MDT0001: Client 082713ee-6e9d-4e5e-ba11-c21407d29a20 (at 192.168.203.53@tcp) reconnecting [ 3521.764784] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3522.374514] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3524.132502] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3524.227598] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3525.544846] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3525.614964] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3526.395703] Lustre: Permanently deactivating lustre-MDT0001 [ 3526.400290] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3526.402483] Lustre: Skipped 2 previous similar messages [ 3528.255918] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3532.259114] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3532.260951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3532.263764] Lustre: Skipped 2 previous similar messages [ 3532.268768] Lustre: Skipped 3 previous similar messages [ 3537.377927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3537.380746] Lustre: Skipped 3 previous similar messages [ 3538.046936] LustreError: 247148:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 3538.074084] Lustre: server umount lustre-MDT0000 complete [ 3539.272619] LustreError: 243042:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759816449 with bad export cookie 11779488176616948034 [ 3539.277724] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3539.352338] LustreError: 247350:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 3539.354406] LustreError: 247350:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3539.428283] Lustre: server umount lustre-MDT0001 complete [ 3546.733199] LustreError: 247550:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 3546.735177] LustreError: 247550:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3546.749436] Lustre: server umount lustre-OST0000 complete [ 3550.298270] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 01:54:20 (1759816460) [ 3553.511597] LustreError: 248555:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3553.513230] LustreError: 248555:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3553.575426] Lustre: server umount lustre-OST0001 complete [ 3555.576863] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 3558.310082] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 3562.061468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3564.310963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3566.264749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3568.323281] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3570.634950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3570.658784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3570.748978] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3570.753789] Lustre: Skipped 2 previous similar messages [ 3570.763750] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3570.808104] Lustre: lustre-MDT0000: new disk, initializing [ 3570.833775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3570.841607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3572.110357] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3576.316247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3576.339935] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3576.372097] Lustre: 250736:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3576.393798] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3576.395437] Lustre: Skipped 1 previous similar message [ 3576.431339] Lustre: lustre-MDT0001: new disk, initializing [ 3576.485415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3576.490239] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3577.824792] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3580.924584] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3582.096836] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3584.380144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3584.406497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3584.514728] Lustre: lustre-OST0000: new disk, initializing [ 3584.517701] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3584.545119] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3584.547088] Lustre: Skipped 1 previous similar message [ 3586.300175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3586.304824] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3586.341280] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3586.423931] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3589.379603] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3590.002061] LustreError: 253090:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3594.169321] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3594.191253] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3594.243268] Lustre: lustre-OST0001: new disk, initializing [ 3594.244934] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3596.203327] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3599.206331] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3600.096075] LustreError: 253090:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3600.102705] LustreError: 253090:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3600.791922] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 3601.406720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3601.409876] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3601.446941] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3610.192081] LustreError: 253090:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3611.619795] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3611.621819] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3611.623291] LustreError: Skipped 1 previous similar message [ 3611.626902] Lustre: Skipped 1 previous similar message [ 3611.627439] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3611.631239] Lustre: Skipped 1 previous similar message [ 3615.233266] Lustre: lustre-OST0001: Not available for connect from 192.168.203.53@tcp (stopping) [ 3616.876112] LustreError: 254229:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3616.877804] LustreError: 254229:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3616.895788] Lustre: server umount lustre-OST0001 complete [ 3621.860912] LustreError: 254142:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3621.864556] LustreError: 254142:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3626.981306] LustreError: 252467:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3626.985585] LustreError: 252467:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3632.099224] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3632.099796] LustreError: 254142:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3632.100758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3632.100764] Lustre: Skipped 2 previous similar messages [ 3632.102193] Lustre: Skipped 2 previous similar messages [ 3632.112905] LustreError: 254142:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3637.220258] LustreError: 252468:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3637.223958] LustreError: 252468:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3642.339311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3642.341642] Lustre: Skipped 4 previous similar messages [ 3643.360121] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3643.380325] LustreError: 254454:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3643.381864] LustreError: 254454:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3643.448113] Lustre: server umount lustre-OST0000 complete [ 3647.457736] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3647.461144] Lustre: Skipped 1 previous similar message [ 3650.911102] Lustre: server umount lustre-MDT0000 complete [ 3652.131056] LustreError: 253485:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759816562 with bad export cookie 11779488176616950841 [ 3652.133691] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3652.135127] LustreError: 253485:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 3666.400096] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3666.430911] LustreError: 254857:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 3666.433839] LustreError: 254857:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 3666.523775] Lustre: server umount lustre-MDT0001 complete [ 3669.963232] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 3671.131790] Key type lgssc unregistered [ 3671.278511] LNet: 255390:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3671.281647] LNetError: 255390:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3671.295450] LNet: Removed LNI 192.168.203.153@tcp [ 3671.616870] Key type .llcrypt unregistered [ 3671.617813] Key type ._llcrypt unregistered [ 3682.361185] Key type ._llcrypt registered [ 3682.362760] Key type .llcrypt registered [ 3682.403224] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 3688.306965] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 3688.667848] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3688.722901] alg: No test for adler32 (adler32-zlib) [ 3689.585118] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 3689.682744] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 3691.264112] Key type lgssc registered [ 3691.602988] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3694.553910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3696.747568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3698.805331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3700.930844] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3703.148944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3703.169264] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3703.174887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3704.256618] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3704.267266] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3704.313115] Lustre: lustre-MDT0000: new disk, initializing [ 3704.336592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3704.342176] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3705.614152] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3709.666729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3709.694561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3709.731714] Lustre: 259311:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3709.757248] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3709.759814] Lustre: Skipped 1 previous similar message [ 3709.802324] Lustre: lustre-MDT0001: new disk, initializing [ 3709.830897] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3709.840935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3709.843493] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3711.012437] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3713.958723] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3715.019742] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3717.137179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3717.161031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3717.255692] Lustre: lustre-OST0000: new disk, initializing [ 3717.259326] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3717.286419] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3719.093876] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3720.252325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3720.257134] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3720.290830] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3722.170816] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3723.579302] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3723.655280] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3725.000650] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3725.086340] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3730.403611] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3730.404459] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3730.406806] LustreError: Skipped 1 previous similar message [ 3730.407474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3730.411593] Lustre: Skipped 1 previous similar message [ 3730.415651] Lustre: Skipped 1 previous similar message [ 3735.522688] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3735.525970] Lustre: Skipped 1 previous similar message [ 3740.128089] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3740.145100] LustreError: 262247:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3740.224755] Lustre: server umount lustre-OST0000 complete [ 3745.761936] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3745.762028] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3745.762780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3745.770065] Lustre: Skipped 1 previous similar message [ 3747.649448] LustreError: 262450:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3747.652585] LustreError: 262450:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3747.686942] Lustre: server umount lustre-MDT0000 complete [ 3748.918157] LustreError: 260090:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759816659 with bad export cookie 13020663820925015091 [ 3748.920848] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3748.923378] LustreError: 260090:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3748.985356] LustreError: 262652:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3748.987123] LustreError: 262652:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3749.085834] Lustre: server umount lustre-MDT0001 complete [ 3751.920447] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 01:57:42 (1759816662) [ 3754.148620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3754.316830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3755.558405] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3758.250192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3758.380913] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3759.747792] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3761.476517] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3762.617953] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3764.727197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3766.681762] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3768.481832] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3824.608810] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3824.610890] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3824.614454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3824.615980] Lustre: Skipped 1 previous similar message [ 3828.333067] LustreError: 265668:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3828.334989] LustreError: 265668:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3828.411679] Lustre: server umount lustre-OST0000 complete [ 3829.873348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 3833.749026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3833.865527] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3833.867253] Lustre: Skipped 1 previous similar message [ 3835.663767] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3837.419580] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3845.094806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3845.100109] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3850.216786] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3850.220119] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3875.811219] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3875.813983] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3875.814517] Lustre: Skipped 1 previous similar message [ 3875.818025] LustreError: Skipped 1 previous similar message [ 3875.820261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3875.822696] Lustre: Skipped 3 previous similar messages [ 3876.909109] LustreError: 267330:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3876.911702] LustreError: 267330:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3876.995736] Lustre: server umount lustre-OST0000 complete [ 3880.930697] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3880.933927] Lustre: Skipped 2 previous similar messages [ 3884.252823] Lustre: server umount lustre-MDT0000 complete [ 3885.374175] LustreError: 265568:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759816795 with bad export cookie 13020663820925015959 [ 3885.378049] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3885.436755] LustreError: 267734:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3885.438279] LustreError: 267734:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 3885.524751] Lustre: server umount lustre-MDT0001 complete [ 3888.830467] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 3889.878179] Key type lgssc unregistered [ 3890.003294] LNet: 268265:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3890.005300] LNetError: 268265:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3890.012395] LNet: Removed LNI 192.168.203.153@tcp [ 3890.309320] Key type .llcrypt unregistered [ 3890.310423] Key type ._llcrypt unregistered [ 3899.210554] Key type ._llcrypt registered [ 3899.211508] Key type .llcrypt registered [ 3899.244415] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 3899.630183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3899.638785] alg: No test for adler32 (adler32-zlib) [ 3900.473663] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 3900.545051] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 3902.120126] Key type lgssc registered [ 3902.442349] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3904.849546] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3904.855532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3906.000218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3907.187624] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3909.902289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3910.027386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3911.265787] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3913.095619] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3914.356842] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3916.507215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3916.634113] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3918.422146] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3920.142797] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3930.985163] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3946.469596] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3946.470197] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3946.471472] LustreError: Skipped 1 previous similar message [ 3946.475371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3946.476821] Lustre: Skipped 1 previous similar message [ 3948.910147] LustreError: 272681:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3948.973412] Lustre: server umount lustre-OST0000 complete [ 3951.072996] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3951.074878] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3951.077636] Lustre: Skipped 1 previous similar message [ 3951.079111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3956.285031] LustreError: 272883:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3956.286783] LustreError: 272883:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3956.314861] Lustre: server umount lustre-MDT0000 complete [ 3956.705457] LustreError: 271007:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3956.709644] LustreError: 271007:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3957.478222] LustreError: 272886:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759816867 with bad export cookie 6062584948455462343 [ 3957.482105] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3957.551084] LustreError: 273098:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3957.552689] LustreError: 273098:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3957.629868] Lustre: server umount lustre-MDT0001 complete [ 3960.964572] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 3962.024373] Key type lgssc unregistered [ 3962.157482] LNet: 273628:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3962.160868] LNetError: 273628:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3962.168374] LNet: Removed LNI 192.168.203.153@tcp [ 3962.475751] Key type .llcrypt unregistered [ 3962.477048] Key type ._llcrypt unregistered [ 3969.043143] Key type ._llcrypt registered [ 3969.044026] Key type .llcrypt registered [ 3969.075434] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 3969.471317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3969.484897] alg: No test for adler32 (adler32-zlib) [ 3970.333346] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 3970.420306] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 3971.992102] Key type lgssc registered [ 3972.282386] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3974.570717] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3974.576990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3975.726625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3976.914560] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3979.625318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3979.743550] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3981.059443] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3982.826812] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3983.947335] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3986.138321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3986.259167] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3988.076246] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 3989.754918] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4000.615631] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4009.952765] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4009.954494] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4009.957490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4010.977726] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4010.977875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4010.978925] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4010.987478] Lustre: Skipped 1 previous similar message [ 4014.829096] LustreError: 277222:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4014.896789] Lustre: server umount lustre-OST0000 complete [ 4016.097188] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4016.097982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4016.100926] Lustre: Skipped 1 previous similar message [ 4021.217927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4021.221293] Lustre: Skipped 2 previous similar messages [ 4022.139810] LustreError: 277423:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4022.141394] LustreError: 277423:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4022.163988] Lustre: server umount lustre-MDT0000 complete [ 4023.241162] LustreError: 275526:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759816933 with bad export cookie 12932955713651105638 [ 4023.243872] LustreError: 275526:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4023.244088] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4023.309850] LustreError: 277624:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4023.311648] LustreError: 277624:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4023.384413] Lustre: server umount lustre-MDT0001 complete [ 4026.605431] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 4027.499288] Key type lgssc unregistered [ 4027.614322] LNet: 278155:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4027.616325] LNetError: 278155:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4027.622295] LNet: Removed LNI 192.168.203.153@tcp [ 4027.879627] Key type .llcrypt unregistered [ 4027.880543] Key type ._llcrypt unregistered [ 4036.599185] Key type ._llcrypt registered [ 4036.600071] Key type .llcrypt registered [ 4036.631572] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 4036.982508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4037.027135] alg: No test for adler32 (adler32-zlib) [ 4037.861949] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 4037.933754] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 4039.504113] Key type lgssc registered [ 4039.804749] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4042.072987] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4042.078575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4043.225527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4044.391161] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 4046.918536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4047.033799] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4048.200683] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 4049.789136] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4050.884285] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4052.923206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4053.040961] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4054.066424] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4054.840733] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 4056.458594] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4067.301368] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4067.304549] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4067.313420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4067.316737] Lustre: Skipped 1 previous similar message [ 4071.155152] LustreError: 282544:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4071.235973] Lustre: server umount lustre-OST0000 complete [ 4072.929566] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4072.937559] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4072.946018] Lustre: Skipped 1 previous similar message [ 4072.950286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4077.537216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4077.537787] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4082.657129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4082.658731] Lustre: Skipped 2 previous similar messages [ 4087.264097] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4087.302462] LustreError: 282750:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4087.304088] LustreError: 282750:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4087.328953] Lustre: server umount lustre-MDT0000 complete [ 4087.777259] LustreError: 280242:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4087.781326] LustreError: 280242:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4088.449224] LustreError: 279509:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759816998 with bad export cookie 14136610519719872877 [ 4088.451919] LustreError: 279509:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4088.452080] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4088.517620] LustreError: 282953:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4088.520071] LustreError: 282953:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4088.606434] Lustre: server umount lustre-MDT0001 complete [ 4091.822196] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 4092.772770] Key type lgssc unregistered [ 4092.903300] LNet: 283485:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4092.905356] LNetError: 283485:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4092.914311] LNet: Removed LNI 192.168.203.153@tcp [ 4093.201965] Key type .llcrypt unregistered [ 4093.202858] Key type ._llcrypt unregistered [ 4099.606737] Key type ._llcrypt registered [ 4099.607721] Key type .llcrypt registered [ 4099.639885] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 4099.921576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4099.944054] alg: No test for adler32 (adler32-zlib) [ 4100.778293] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 4100.850306] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 4102.424105] Key type lgssc registered [ 4102.755987] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4105.107176] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4105.122010] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4106.351320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4107.886786] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 4111.805391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4111.996835] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4113.412236] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 4115.624696] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4117.111513] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4119.589672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4119.761047] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4122.001192] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 4124.091641] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4131.048059] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 4138.465177] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4138.467630] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4138.472632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4141.538187] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4141.539851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4141.540235] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4143.916223] LustreError: 287121:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4143.973775] Lustre: server umount lustre-OST0000 complete [ 4146.658522] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4146.660371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4146.664478] Lustre: Skipped 1 previous similar message [ 4146.669878] Lustre: Skipped 2 previous similar messages [ 4151.229857] LustreError: 287324:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4151.231471] LustreError: 287324:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4151.259708] Lustre: server umount lustre-MDT0000 complete [ 4151.777064] LustreError: 284694:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4151.782015] LustreError: 284694:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4152.399338] LustreError: 285425:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759817062 with bad export cookie 12291240655792543389 [ 4152.402957] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4152.404297] LustreError: 285425:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4152.463627] LustreError: 287525:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4152.466021] LustreError: 287525:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4152.550937] Lustre: server umount lustre-MDT0001 complete [ 4155.809953] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 4156.742072] Key type lgssc unregistered [ 4156.886501] LNet: 288055:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4156.890107] LNetError: 288055:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4156.898592] LNet: Removed LNI 192.168.203.153@tcp [ 4157.364068] Key type .llcrypt unregistered [ 4157.366119] Key type ._llcrypt unregistered [ 4161.416129] Key type ._llcrypt registered [ 4161.417156] Key type .llcrypt registered [ 4161.449303] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4166.453432] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 4172.308953] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 4172.555514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4172.567982] alg: No test for adler32 (adler32-zlib) [ 4173.404987] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 4173.478217] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 4175.056090] Key type lgssc registered [ 4175.378184] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4178.069181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4180.212511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4182.017393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4183.935752] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4185.992988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4186.009312] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4186.015377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4187.092588] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4187.102471] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4187.137384] Lustre: lustre-MDT0000: new disk, initializing [ 4187.159107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4187.164342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4188.291434] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 4192.236250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4192.261792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4192.302236] Lustre: 291538:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4192.321862] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4192.323937] Lustre: Skipped 1 previous similar message [ 4192.359536] Lustre: lustre-MDT0001: new disk, initializing [ 4192.390656] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4192.401229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4192.405630] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4193.597734] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 4196.513122] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4197.568875] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4199.634043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4199.656716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4199.740773] Lustre: lustre-OST0000: new disk, initializing [ 4199.742384] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4199.766096] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4201.504304] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 4202.041373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4202.044677] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4202.076635] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4204.421749] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4205.610687] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4205.679785] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4206.790317] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4206.857094] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4209.518167] LustreError: 294471:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4209.578407] Lustre: server umount lustre-OST0000 complete [ 4212.193840] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4212.194367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4212.198436] Lustre: Skipped 1 previous similar message [ 4217.313704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4217.317477] Lustre: Skipped 2 previous similar messages [ 4222.433144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4222.435045] Lustre: Skipped 1 previous similar message [ 4224.992107] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4225.031668] LustreError: 294673:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4225.033932] LustreError: 294673:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4225.066591] Lustre: server umount lustre-MDT0000 complete [ 4226.635302] LustreError: 291529:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759817136 with bad export cookie 15284757066306948804 [ 4226.638738] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4226.641914] LustreError: 291529:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4236.771695] LustreError: 294875:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4236.774216] LustreError: 294875:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4236.844091] Lustre: server umount lustre-MDT0001 complete [ 4239.407197] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 02:05:49 (1759817149) [ 4241.506849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4241.669539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4242.890772] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 4245.643195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4245.753622] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4246.979198] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 4248.630224] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4249.878992] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4252.034196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4253.919551] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 4255.680820] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4260.323393] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4260.324027] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4260.330560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4260.333267] Lustre: Skipped 1 previous similar message [ 4265.386545] LustreError: 297787:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4265.388155] LustreError: 297787:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4265.446369] Lustre: server umount lustre-OST0000 complete [ 4270.561649] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4270.562155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4270.564812] Lustre: Skipped 2 previous similar messages [ 4270.567310] Lustre: Skipped 1 previous similar message [ 4272.764084] LustreError: 297993:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4272.766691] LustreError: 297993:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4272.790853] Lustre: server umount lustre-MDT0000 complete [ 4273.935946] LustreError: 295527:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759817184 with bad export cookie 15284757066306949672 [ 4273.938759] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4273.941115] LustreError: 295527:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4274.085603] Lustre: server umount lustre-MDT0001 complete [ 4277.381063] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 4278.449326] Key type lgssc unregistered [ 4278.595464] LNet: 298724:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4278.597499] LNetError: 298724:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4278.608492] LNet: Removed LNI 192.168.203.153@tcp [ 4278.947728] Key type .llcrypt unregistered [ 4278.948818] Key type ._llcrypt unregistered [ 4286.707610] Key type ._llcrypt registered [ 4286.708466] Key type .llcrypt registered [ 4286.742255] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 4293.784899] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 4294.163731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4294.188593] alg: No test for adler32 (adler32-zlib) [ 4295.051162] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 4295.123875] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 4296.704181] Key type lgssc registered [ 4297.205767] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4301.399960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4304.275717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4306.678788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4309.217002] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4311.359098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4313.970895] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4317.218452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4317.243242] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4317.253420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4318.351733] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4318.366668] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4318.400937] Lustre: lustre-MDT0000: new disk, initializing [ 4318.435561] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4318.443332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4320.062646] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 4325.014222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4325.042370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4325.076011] Lustre: 302502:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4325.101139] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4325.104469] Lustre: Skipped 1 previous similar message [ 4325.146728] Lustre: lustre-MDT0001: new disk, initializing [ 4325.188525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4325.206107] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4325.210875] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4326.847766] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 4330.467131] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4332.411090] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4335.907769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4335.950408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4336.111979] Lustre: lustre-OST2710: new disk, initializing [ 4336.114886] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4336.157870] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4338.868264] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 4342.692490] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 4345.809259] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4345.847850] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4345.921313] Lustre: lustre-OST03e8: new disk, initializing [ 4345.925816] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4345.978373] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4346.041158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4346.044646] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4346.047195] Lustre: Skipped 1 previous similar message [ 4346.119430] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4346.126454] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4348.329890] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 4350.735351] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 4353.923638] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4354.023247] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4355.689132] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4355.768526] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4357.067129] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4357.140498] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4358.327808] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4358.402919] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4361.185092] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4361.186936] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4361.190333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4361.702274] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4361.705730] Lustre: Skipped 2 previous similar messages [ 4361.705920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4361.708518] Lustre: Skipped 2 previous similar messages [ 4365.500305] LustreError: 306842:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 4365.530100] Lustre: server umount lustre-MDT0000 complete [ 4366.661258] LustreError: 302493:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759817276 with bad export cookie 10411781167730497147 [ 4366.665245] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4366.667449] LustreError: 302493:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4366.737385] LustreError: 307044:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 4366.740259] LustreError: 307044:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4366.830087] Lustre: server umount lustre-MDT0001 complete [ 4378.226667] LustreError: 307244:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4378.228583] LustreError: 307244:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4378.246882] Lustre: server umount lustre-OST2710 complete [ 4389.617971] LustreError: 307445:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4389.619620] LustreError: 307445:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4389.681626] Lustre: server umount lustre-OST03e8 complete [ 4395.407824] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 4398.615925] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 4401.795535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4403.832959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4406.118342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4408.622825] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4410.708066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4410.730367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4410.809567] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4410.819499] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4410.854521] Lustre: lustre-MDT0000: new disk, initializing [ 4410.876694] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4410.882626] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4410.886449] Lustre: Skipped 1 previous similar message [ 4412.067958] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 4415.899912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4415.921304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4415.951289] Lustre: 310238:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4416.040716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4416.043379] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4417.169745] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 4420.041621] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4421.111837] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4423.171784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4423.191031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4423.278340] Lustre: lustre-OST0000: new disk, initializing [ 4423.280283] Lustre: Skipped 1 previous similar message [ 4423.282532] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4423.285461] Lustre: Skipped 2 previous similar messages [ 4423.311219] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4423.314080] Lustre: Skipped 1 previous similar message [ 4424.958709] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 4427.808196] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4429.052939] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4430.328386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4430.332784] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4430.365074] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4431.163068] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 4432.376554] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4432.442752] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4435.428252] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4435.428413] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4435.431773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4435.432867] LustreError: Skipped 1 previous similar message [ 4435.435781] Lustre: Skipped 1 previous similar message [ 4439.149089] LustreError: 313190:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4439.151399] LustreError: 313190:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4439.213326] Lustre: server umount lustre-OST0000 complete [ 4440.545982] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4440.546851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4440.554063] Lustre: Skipped 2 previous similar messages [ 4445.665104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4445.667292] Lustre: Skipped 2 previous similar messages [ 4446.555240] Lustre: server umount lustre-MDT0000 complete [ 4447.738324] LustreError: 310228:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759817357 with bad export cookie 10411781167730498715 [ 4447.741488] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4447.743329] LustreError: 310228:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4447.807531] LustreError: 313596:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4447.810362] LustreError: 313596:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 4447.886200] Lustre: server umount lustre-MDT0001 complete [ 4450.629648] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 02:09:20 (1759817360) [ 4451.095380] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4451.701967] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 02:09:21 (1759817361) [ 4452.616204] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 4455.505928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4457.108262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4458.412930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4459.630112] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4459.966655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4462.627610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4462.774717] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4462.786964] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4462.873691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4464.389086] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 4467.111654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4467.140531] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4467.151369] Lustre: 315725:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4468.455515] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 4470.221845] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4471.418097] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4473.480107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4473.503155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4473.572084] Lustre: Denying initial registration attempt from nid 192.168.203.153@tcp, specified as failover [ 4473.574540] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4473.579294] LustreError: 317342:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -99 [ 4473.582685] LustreError: 317342:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 4473.585632] LustreError: 317342:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 4473.642708] Lustre: server umount lustre-OST0000 complete [ 4473.643921] LustreError: 317342:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4477.408920] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4477.412421] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4477.418453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4481.789074] LustreError: 317709:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 4481.790833] LustreError: 317709:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4481.808816] Lustre: server umount lustre-MDT0000 complete [ 4482.948039] LustreError: 316452:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759817393 with bad export cookie 10411781167730499583 [ 4482.950880] LustreError: 316452:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4482.951453] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4483.080619] Lustre: server umount lustre-MDT0001 complete [ 4486.376679] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 4487.342505] Key type lgssc unregistered [ 4487.462369] LNet: 318441:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4487.466114] LNetError: 318441:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4487.475306] LNet: Removed LNI 192.168.203.153@tcp [ 4487.758962] Key type .llcrypt unregistered [ 4487.760511] Key type ._llcrypt unregistered [ 4490.949471] Key type ._llcrypt registered [ 4490.950397] Key type .llcrypt registered [ 4490.982275] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 4491.263810] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4491.270209] alg: No test for adler32 (adler32-zlib) [ 4492.104850] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 4492.176579] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 4493.752131] Key type lgssc registered [ 4494.055872] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4496.154609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4497.283123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4498.414413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4499.560687] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4499.874161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4505.778800] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 4508.401918] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4508.409734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4509.498160] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4509.505479] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4509.560756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4510.761135] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 4513.405733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4513.435961] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4513.446060] Lustre: 321172:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4513.543118] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4514.741126] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 4516.394842] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4517.581291] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4520.640912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4520.681327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4520.790467] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4520.900516] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4523.219619] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 4525.455238] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4529.124190] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4529.128710] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4529.130200] LustreError: Skipped 1 previous similar message [ 4529.141096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4529.148726] Lustre: Skipped 1 previous similar message [ 4532.593870] LustreError: 323457:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4532.693168] Lustre: server umount lustre-OST0000 complete [ 4539.363700] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4539.363848] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4539.365280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4539.378416] Lustre: Skipped 2 previous similar messages [ 4540.620295] LustreError: 323660:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4540.625260] LustreError: 323660:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4540.659958] Lustre: server umount lustre-MDT0000 complete [ 4542.205187] LustreError: 321900:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759817452 with bad export cookie 17526926712071810573 [ 4542.207646] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4542.208059] LustreError: 321900:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4542.278977] LustreError: 323863:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4542.280589] LustreError: 323863:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4542.357375] Lustre: server umount lustre-MDT0001 complete [ 4547.009615] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 4548.314988] Key type lgssc unregistered [ 4548.495703] LNet: 324394:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4548.499849] LNetError: 324394:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4548.512559] LNet: Removed LNI 192.168.203.153@tcp [ 4548.907656] Key type .llcrypt unregistered [ 4548.908911] Key type ._llcrypt unregistered [ 4560.408793] Key type ._llcrypt registered [ 4560.410847] Key type .llcrypt registered [ 4560.473369] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 4560.814226] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4560.862425] alg: No test for adler32 (adler32-zlib) [ 4561.714248] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 4561.795882] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 4563.384169] Key type lgssc registered [ 4563.967335] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4567.713106] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4567.724782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4568.940478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4570.609874] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 4574.667431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4574.818928] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4576.521453] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 4578.925881] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4580.495254] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4583.606152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4583.773789] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4586.181075] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 4588.718140] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4599.269525] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4599.271907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4599.276984] Lustre: Skipped 2 previous similar messages [ 4599.285537] Lustre: Skipped 2 previous similar messages [ 4602.555452] LustreError: 328017:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4602.583274] Lustre: server umount lustre-MDT0000 complete [ 4603.690052] LustreError: 325749:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759817513 with bad export cookie 2093627544217619825 [ 4603.693032] LustreError: 325749:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4603.693345] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4604.384833] LustreError: 327927:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4604.385118] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4604.390561] LustreError: 327927:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4604.394909] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4604.398092] Lustre: Skipped 1 previous similar message [ 4608.416766] LustreError: 325763:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4608.417065] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4608.420826] LustreError: 325763:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4608.422317] Lustre: Skipped 1 previous similar message [ 4609.527392] LustreError: 325763:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4609.788673] LustreError: 328219:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4609.790402] LustreError: 328219:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4609.860769] Lustre: server umount lustre-MDT0001 complete [ 4611.784897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4614.324617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4627.936523] LustreError: 325078:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000006f8823f0 x1845302364494336/t0(0) o250->MGC192.168.203.153@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4628.034357] LustreError: 327379:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4628.056118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4628.075033] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 4629.142560] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 4631.597619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4632.812975] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 4633.764966] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4634.309387] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4638.884690] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4640.835292] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 6 sec [ 4641.694023] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4642.099673] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4643.296605] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4643.298166] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4643.300799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4643.302021] Lustre: Skipped 1 previous similar message [ 4649.197069] LustreError: 330519:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4649.198387] LustreError: 330519:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4649.246745] Lustre: server umount lustre-OST0000 complete [ 4652.512760] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4652.514337] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4652.516690] Lustre: Skipped 1 previous similar message [ 4652.517873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4652.519022] Lustre: Skipped 2 previous similar messages [ 4656.312450] LustreError: 330721:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4656.315115] LustreError: 330721:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4656.337290] Lustre: server umount lustre-MDT0000 complete [ 4657.376179] LustreError: 327389:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759817567 with bad export cookie 2093627544217626986 [ 4657.378615] LustreError: 327389:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4657.378628] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4657.508440] Lustre: server umount lustre-MDT0001 complete [ 4661.056599] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 4661.986386] Key type lgssc unregistered [ 4662.113274] LNet: 331455:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4662.115344] LNetError: 331455:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4662.122331] LNet: Removed LNI 192.168.203.153@tcp [ 4662.422068] Key type .llcrypt unregistered [ 4662.422973] Key type ._llcrypt unregistered [ 4670.435820] Key type ._llcrypt registered [ 4670.437362] Key type .llcrypt registered [ 4670.469986] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 4670.757183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4670.784078] alg: No test for adler32 (adler32-zlib) [ 4671.621948] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 4671.694061] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 4673.272108] Key type lgssc registered [ 4673.570220] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4675.813872] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4675.819738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4676.961429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4678.066950] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 4680.638186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4680.757281] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4681.907551] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 4683.664466] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4685.012184] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4686.074242] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4686.077799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4686.079111] Lustre: Skipped 1 previous similar message [ 4690.210494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4690.213862] Lustre: Skipped 1 previous similar message [ 4692.218194] LustreError: 334219:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4692.242363] Lustre: server umount lustre-MDT0000 complete [ 4693.385310] LustreError: 332808:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759817603 with bad export cookie 782126869356908030 [ 4693.388203] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4693.388403] LustreError: 332808:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4693.467073] LustreError: 334420:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4693.469081] LustreError: 334420:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4693.560469] Lustre: server umount lustre-MDT0001 complete [ 4699.683305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4699.789372] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4699.803610] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4699.890742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4701.049178] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 4704.743899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4704.796813] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4704.814711] Lustre: 335191:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4704.947697] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4706.146662] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 4708.404580] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4710.055500] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4713.208713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4713.319756] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 4713.325919] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 4713.331304] LustreError: 336852:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 4713.335500] LustreError: 336852:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 4713.339206] LustreError: 336852:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 4713.415541] Lustre: server umount lustre-OST0000 complete [ 4713.417758] LustreError: 336852:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4717.202643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4717.322243] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4717.419743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4719.091732] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 4719.882935] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 4721.506192] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4723.866636] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4723.891877] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4723.912299] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4723.940045] Lustre: lustre-OST0001: new disk, initializing [ 4723.941596] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4726.222416] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 4728.674856] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4729.373917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4729.380220] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4729.434916] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4734.440549] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4734.441848] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4734.444723] LustreError: Skipped 1 previous similar message [ 4734.453970] Lustre: Skipped 1 previous similar message [ 4734.455156] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4734.465158] Lustre: Skipped 1 previous similar message [ 4736.051153] LustreError: 338969:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 4736.054692] LustreError: 338969:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4736.077303] Lustre: server umount lustre-OST0001 complete [ 4739.559290] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4739.559904] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4739.560160] LustreError: 337526:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4739.560226] LustreError: 337526:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4739.568673] Lustre: Skipped 1 previous similar message [ 4739.573748] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4739.594931] Lustre: Skipped 1 previous similar message [ 4744.675278] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4744.676086] LustreError: 337321:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4744.677357] Lustre: Skipped 1 previous similar message [ 4744.680942] LustreError: 337321:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4749.798394] LustreError: 339172:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4749.804500] LustreError: 339172:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4752.352168] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4752.373076] LustreError: 339170:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4752.375359] LustreError: 339170:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4752.440560] Lustre: server umount lustre-OST0000 complete [ 4754.914829] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4754.916398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4754.925047] Lustre: Skipped 1 previous similar message [ 4754.928429] Lustre: Skipped 2 previous similar messages [ 4760.256725] LustreError: 339375:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4760.260091] LustreError: 339375:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4760.292239] Lustre: server umount lustre-MDT0000 complete [ 4762.092902] LustreError: 335916:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759817672 with bad export cookie 782126869356908569 [ 4762.099065] LustreError: 335916:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4762.099732] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4762.272237] Lustre: server umount lustre-MDT0001 complete [ 4766.724505] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 4768.255984] Key type lgssc unregistered [ 4768.452764] LNet: 340110:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4768.457529] LNetError: 340110:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4768.467738] LNet: Removed LNI 192.168.203.153@tcp [ 4768.990617] Key type .llcrypt unregistered [ 4768.992405] Key type ._llcrypt unregistered [ 4784.894789] Key type ._llcrypt registered [ 4784.896679] Key type .llcrypt registered [ 4784.964376] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 4790.746447] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 4791.328876] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4791.340298] alg: No test for adler32 (adler32-zlib) [ 4792.184285] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 4792.270301] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 4793.848182] Key type lgssc registered [ 4794.235208] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4798.611929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4801.932848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4804.700634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4807.671742] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4810.979757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4811.005370] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4811.013226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4812.131792] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4812.152192] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4812.198836] Lustre: lustre-MDT0000: new disk, initializing [ 4812.239196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4812.250039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4814.109591] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 4819.837895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4819.867923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4819.915621] Lustre: 344032:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4819.947802] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4819.950490] Lustre: Skipped 1 previous similar message [ 4819.995345] Lustre: lustre-MDT0001: new disk, initializing [ 4820.038592] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4820.059890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4820.066375] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4821.873809] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 4826.008459] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4827.745723] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4830.909223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4830.943422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4831.066402] Lustre: lustre-OST0000: new disk, initializing [ 4831.072441] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4831.120818] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4831.256345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4831.260744] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4831.316621] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4833.783879] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 4837.507831] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4839.109795] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4839.210930] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4840.645391] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4840.733933] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4841.443874] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4841.444680] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4841.445614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4841.447679] Lustre: Skipped 1 previous similar message [ 4841.449393] LustreError: Skipped 1 previous similar message [ 4846.562159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4846.564641] Lustre: Skipped 2 previous similar messages [ 4847.470143] LustreError: 346971:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4847.534917] Lustre: server umount lustre-OST0000 complete [ 4850.657176] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4850.660530] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4850.666306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4851.681691] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4854.913520] LustreError: 347172:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4854.915578] LustreError: 347172:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4854.945274] Lustre: server umount lustre-MDT0000 complete [ 4856.349320] LustreError: 344810:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759817766 with bad export cookie 6367614254355104047 [ 4856.352064] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4856.353653] LustreError: 344810:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4856.421491] LustreError: 347374:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4856.424065] LustreError: 347374:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4856.512842] Lustre: server umount lustre-MDT0001 complete [ 4859.436372] Lustre: DEBUG MARKER: == conf-sanity test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 02:16:09 (1759817769) [ 4861.217258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4863.067203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4870.658407] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 4873.848568] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 4877.467467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4880.015519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4882.268419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4884.461256] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4886.848388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4886.872495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4886.956375] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4886.967552] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4887.004563] Lustre: lustre-MDT0000: new disk, initializing [ 4887.029352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4887.035674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4888.415896] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 4892.576580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4892.601925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4892.635397] Lustre: 351139:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4892.658126] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4892.660227] Lustre: Skipped 1 previous similar message [ 4892.696389] Lustre: lustre-MDT0001: new disk, initializing [ 4892.732358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4892.744619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4892.749063] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4894.089786] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 4897.173946] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4898.313429] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4900.547052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4900.569634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4902.569749] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 4905.501675] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4906.751782] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4906.997195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4906.999709] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4907.026530] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4907.835984] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 4909.075712] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4909.145346] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4912.099145] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4912.099394] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4912.100496] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4912.100501] Lustre: Skipped 2 previous similar messages [ 4912.101822] LustreError: Skipped 1 previous similar message [ 4912.104870] Lustre: Skipped 1 previous similar message [ 4915.823750] LustreError: 354086:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4915.827285] LustreError: 354086:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4915.896811] Lustre: server umount lustre-OST0000 complete [ 4918.240797] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4918.243684] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4918.248359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4918.250642] Lustre: Skipped 1 previous similar message [ 4923.388416] LustreError: 354287:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4923.390910] LustreError: 354287:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4923.415579] Lustre: server umount lustre-MDT0000 complete [ 4924.657341] LustreError: 351132:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759817834 with bad export cookie 6367614254355104929 [ 4924.658796] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4924.662906] LustreError: 351132:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4924.801850] Lustre: server umount lustre-MDT0001 complete [ 4927.622838] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 02:17:17 (1759817837) [ 4930.348825] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 02:17:20 (1759817840) [ 4933.809383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4934.032152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4934.035351] Lustre: Skipped 1 previous similar message [ 4935.528112] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 4938.709859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4940.281541] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 4942.309939] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4943.716524] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4946.299837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4948.638888] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 4950.585226] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4964.844467] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 4965.421127] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4965.978442] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4966.516833] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 4967.138050] Lustre: Failing over lustre-MDT0000 [ 4967.251120] LustreError: 357963:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4967.252979] LustreError: 357963:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 4967.278112] Lustre: server umount lustre-MDT0000 complete [ 4969.440818] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4969.442892] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4969.445909] Lustre: Skipped 1 previous similar message [ 4969.447406] LustreError: 355505:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4969.453599] LustreError: 355505:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4969.954360] LustreError: 355500:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4969.957869] LustreError: 355500:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4971.517076] LustreError: 356760:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4975.074040] LustreError: 355523:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4975.081021] LustreError: 355523:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4980.108162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4980.144725] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4980.193279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 4980.196317] Lustre: Skipped 4 previous similar messages [ 4980.229125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4980.231423] Lustre: Skipped 2 previous similar messages [ 4980.255976] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4981.587773] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 4981.751468] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4985.319399] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4985.326178] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4985.340183] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4986.428966] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4986.934989] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4988.728240] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 4989.318586] Lustre: Failing over lustre-MDT0000 [ 4989.468901] LustreError: 359043:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4989.470585] LustreError: 359043:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4989.495596] Lustre: server umount lustre-MDT0000 complete [ 4990.435980] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4990.437781] LustreError: 355505:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4990.440668] Lustre: Skipped 4 previous similar messages [ 4990.440836] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4990.444491] LustreError: 355505:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5000.675474] LustreError: 355505:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5000.680302] LustreError: 355505:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 5002.328160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5002.369957] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5002.491268] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5003.857374] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 5007.351767] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5007.853423] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5007.855054] Lustre: Skipped 2 previous similar messages [ 5007.861949] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5007.878437] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5009.055138] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5009.591906] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5012.961606] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5012.963920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5012.966969] Lustre: Skipped 2 previous similar messages [ 5018.005558] Lustre: server umount lustre-MDT0000 complete [ 5018.081622] LustreError: 357121:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5018.087419] LustreError: 357121:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5019.292600] LustreError: 360075:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759817929 with bad export cookie 6367614254355107596 [ 5019.296152] LustreError: 360075:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5019.297397] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5019.468711] Lustre: server umount lustre-MDT0001 complete [ 5030.003476] LustreError: 360476:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5030.005098] LustreError: 360476:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 5030.081458] Lustre: server umount lustre-OST0000 complete [ 5035.072487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5035.251547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5035.253332] Lustre: Skipped 1 previous similar message [ 5036.521571] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 5039.329211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5040.740974] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 5042.443215] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5043.644571] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5045.842835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5047.812785] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 5049.609101] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5060.326556] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5069.792931] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5069.795113] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5069.798141] Lustre: Skipped 3 previous similar messages [ 5069.800623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5072.551174] Lustre: server umount lustre-OST0000 complete [ 5088.224110] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5088.291700] Lustre: server umount lustre-MDT0000 complete [ 5089.616248] LustreError: 362023:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759817999 with bad export cookie 6367614254355108303 [ 5089.619636] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5089.623236] LustreError: 362023:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5089.791243] Lustre: server umount lustre-MDT0001 complete [ 5093.429191] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 5094.551958] Key type lgssc unregistered [ 5094.690322] LNet: 364514:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5094.692599] LNetError: 364514:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5094.699293] LNet: Removed LNI 192.168.203.153@tcp [ 5094.998741] Key type .llcrypt unregistered [ 5094.999849] Key type ._llcrypt unregistered [ 5104.974038] Key type ._llcrypt registered [ 5104.975625] Key type .llcrypt registered [ 5105.013962] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 5105.484954] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5105.492558] alg: No test for adler32 (adler32-zlib) [ 5106.355079] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 5106.445742] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 5108.032144] Key type lgssc registered [ 5108.467454] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5111.030494] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5111.038882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5112.186455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5113.451636] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 5116.077496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5116.182594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5117.498349] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 5119.347543] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5120.609081] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5122.978738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5123.091050] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5124.986537] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 5126.808645] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5137.448032] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5147.105179] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5147.107235] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5147.111361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5147.618197] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5147.619034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5147.623022] Lustre: Skipped 1 previous similar message [ 5152.511002] LustreError: 368258:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5152.541834] Lustre: server umount lustre-MDT0000 complete [ 5152.738436] LustreError: 365983:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5152.742906] LustreError: 365983:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5153.797321] LustreError: 366014:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759818063 with bad export cookie 11379231771154542611 [ 5153.800564] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5153.801035] LustreError: 366014:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5153.863641] LustreError: 368460:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5153.865315] LustreError: 368460:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5153.934654] Lustre: server umount lustre-MDT0001 complete [ 5165.367231] LustreError: 368662:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5165.370059] LustreError: 368662:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5165.439282] Lustre: server umount lustre-OST0000 complete [ 5170.010381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5170.169141] LustreError: 369391:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5170.191620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5171.792770] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 5174.667102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5174.840094] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5176.197153] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 5177.937085] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5179.194223] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5181.475753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5183.487926] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 5185.274419] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5194.924454] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5205.474203] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5205.474518] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5205.476932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5205.479823] Lustre: Skipped 2 previous similar messages [ 5205.483822] Lustre: Skipped 3 previous similar messages [ 5208.572742] LustreError: 371759:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5208.575762] LustreError: 371759:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5208.604411] Lustre: server umount lustre-MDT0000 complete [ 5209.807208] LustreError: 369416:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759818119 with bad export cookie 11379231771154543710 [ 5209.810611] LustreError: 369416:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5209.811086] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5209.953993] Lustre: server umount lustre-MDT0001 complete [ 5221.428575] LustreError: 372163:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5221.431496] LustreError: 372163:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 5221.500813] Lustre: server umount lustre-OST0000 complete [ 5226.605478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5226.767786] LustreError: 372992:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5226.793141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5226.796658] Lustre: Skipped 1 previous similar message [ 5228.051215] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 5230.858718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5232.277302] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 5233.999812] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5235.195126] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5237.353675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5239.356188] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 5241.100472] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5251.815402] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5262.305417] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5262.308336] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5262.308853] LustreError: Skipped 1 previous similar message [ 5262.312806] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5262.315428] Lustre: Skipped 1 previous similar message [ 5263.916102] LustreError: 375259:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5263.918863] LustreError: 375259:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5263.979129] Lustre: server umount lustre-OST0000 complete [ 5266.913174] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5266.915039] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5266.917803] Lustre: Skipped 1 previous similar message [ 5266.919419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5271.324683] Lustre: server umount lustre-MDT0000 complete [ 5272.545447] LustreError: 372991:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5272.550447] LustreError: 372991:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5272.597329] LustreError: 372971:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759818182 with bad export cookie 11379231771154544718 [ 5272.600775] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5272.601153] LustreError: 372971:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5272.762445] Lustre: server umount lustre-MDT0001 complete [ 5276.373318] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 5277.501409] Key type lgssc unregistered [ 5277.647387] LNet: 376196:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5277.649418] LNetError: 376196:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5277.656433] LNet: Removed LNI 192.168.203.153@tcp [ 5278.017581] Key type .llcrypt unregistered [ 5278.018705] Key type ._llcrypt unregistered [ 5287.284667] Key type ._llcrypt registered [ 5287.285563] Key type .llcrypt registered [ 5287.330308] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 5287.618134] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5287.624951] alg: No test for adler32 (adler32-zlib) [ 5288.465464] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 5288.540908] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 5290.120171] Key type lgssc registered [ 5290.463690] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5293.011280] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5293.016766] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5293.019057] LustreError: 377593:0:(osd_handler.c:8486:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5293.025937] LustreError: 377593:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5293.028158] LustreError: 377593:0:(obd_mount.c:193:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5293.030055] LustreError: 377593:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5293.032298] LustreError: 377593:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5295.623031] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5295.625669] LustreError: 377990:0:(osd_handler.c:8486:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5295.634348] LustreError: 377990:0:(obd_config.c:783:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5295.636716] LustreError: 377990:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5295.638746] LustreError: 377990:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5295.640990] LustreError: 377990:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5302.064848] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 5303.268355] Key type lgssc unregistered [ 5303.401372] LNet: 378973:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5303.403429] LNetError: 378973:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5303.415396] LNet: Removed LNI 192.168.203.153@tcp [ 5303.746339] Key type .llcrypt unregistered [ 5303.747249] Key type ._llcrypt unregistered [ 5309.543774] Key type ._llcrypt registered [ 5309.544741] Key type .llcrypt registered [ 5309.583432] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 5315.667677] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 5316.107200] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5316.130620] alg: No test for adler32 (adler32-zlib) [ 5316.983146] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 5317.075902] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 5318.664154] Key type lgssc registered [ 5319.018092] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5322.118915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5324.380354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5326.397373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5328.450986] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5330.773906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5330.793983] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5330.799159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5331.900821] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5331.913034] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5331.955364] Lustre: lustre-MDT0000: new disk, initializing [ 5331.980853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5331.988750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5333.324320] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 5337.365899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5337.391869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5337.432805] Lustre: 382077:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5337.461229] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5337.462950] Lustre: Skipped 1 previous similar message [ 5337.504630] Lustre: lustre-MDT0001: new disk, initializing [ 5337.539221] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5337.551103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5337.554281] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5338.852124] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 5341.778733] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5342.940928] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5345.169811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5345.204145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5345.327604] Lustre: lustre-OST0000: new disk, initializing [ 5345.330889] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5345.369349] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5346.819914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5346.824706] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5346.860101] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5347.505611] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 5350.702282] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5352.028338] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5352.100579] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5353.388276] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5353.463122] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5356.141950] LustreError: 385015:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5356.205780] Lustre: server umount lustre-OST0000 complete [ 5358.048930] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5358.051553] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5358.058440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5362.145626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5362.145918] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5367.265168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5367.266882] Lustre: Skipped 2 previous similar messages [ 5367.268707] LustreError: 385216:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5367.270380] LustreError: 385216:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5367.293591] Lustre: server umount lustre-MDT0000 complete [ 5368.498094] LustreError: 383824:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759818278 with bad export cookie 16347301577241390792 [ 5368.501903] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5368.503632] LustreError: 383824:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5378.019853] LustreError: 385418:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5378.022471] LustreError: 385418:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5378.105589] Lustre: server umount lustre-MDT0001 complete [ 5380.778577] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5381.391191] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 02:24:51 (1759818291) [ 5384.225919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5384.460756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5385.832225] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 5388.606091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5388.735679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5390.008744] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 5391.788106] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5392.979123] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5395.377139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5397.441448] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 5399.328989] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5401.679977] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5401.708072] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5401.762063] Lustre: lustre-OST0001: new disk, initializing [ 5401.763654] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5401.791814] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5401.793384] Lustre: Skipped 1 previous similar message [ 5403.656180] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 5403.841052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5403.845371] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5403.886381] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5406.716096] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5409.254558] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5409.254707] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5409.256921] LustreError: Skipped 1 previous similar message [ 5409.257561] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5409.260742] Lustre: Skipped 1 previous similar message [ 5413.378950] Lustre: lustre-OST0001: Not available for connect from 192.168.203.53@tcp (stopping) [ 5413.381218] Lustre: Skipped 3 previous similar messages [ 5414.701103] LustreError: 389339:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5414.702864] LustreError: 389339:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5414.720512] Lustre: server umount lustre-OST0001 complete [ 5415.396168] LustreError: 387738:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5415.399863] LustreError: 387738:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5420.516450] LustreError: 387738:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5420.521316] LustreError: 387738:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5425.637383] LustreError: 387738:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5425.645077] LustreError: 387738:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5429.216901] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5429.219156] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5429.224688] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5430.756796] LustreError: 387736:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5430.761435] LustreError: 387736:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5432.749983] LustreError: 389562:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5432.752280] LustreError: 389562:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5432.817403] Lustre: server umount lustre-OST0000 complete [ 5434.849194] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5434.851859] LustreError: Skipped 1 previous similar message [ 5434.853027] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5434.856042] Lustre: Skipped 1 previous similar message [ 5440.288987] Lustre: server umount lustre-MDT0000 complete [ 5440.992910] LustreError: 386159:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5441.000069] LustreError: 386159:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5441.514064] LustreError: 386122:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759818351 with bad export cookie 16347301577241391660 [ 5441.517030] LustreError: 386122:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5441.517085] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5447.618189] LustreError: 389965:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 5447.619875] LustreError: 389965:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 5447.697622] Lustre: server umount lustre-MDT0001 complete [ 5451.455535] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 5452.602046] Key type lgssc unregistered [ 5452.744286] LNet: 390496:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5452.746674] LNetError: 390496:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5452.755412] LNet: Removed LNI 192.168.203.153@tcp [ 5453.077936] Key type .llcrypt unregistered [ 5453.079981] Key type ._llcrypt unregistered [ 5464.618499] Key type ._llcrypt registered [ 5464.619394] Key type .llcrypt registered [ 5464.652962] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 5471.009151] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 5471.419264] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5471.437573] alg: No test for adler32 (adler32-zlib) [ 5472.289817] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 5472.368765] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 5473.944170] Key type lgssc registered [ 5474.308613] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5477.851958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5480.290879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5482.189526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5484.137424] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5486.316084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5486.340571] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5486.348725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5487.432539] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5487.444140] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5487.482707] Lustre: lustre-MDT0000: new disk, initializing [ 5487.507582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5487.515197] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5488.803499] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 5493.107495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5493.127822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5493.158418] Lustre: 394417:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5493.181559] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5493.183402] Lustre: Skipped 1 previous similar message [ 5493.224407] Lustre: lustre-MDT0001: new disk, initializing [ 5493.256519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5493.268301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5493.272746] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5494.580220] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 5497.706987] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5498.938699] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5501.333626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5501.366788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5501.486212] Lustre: lustre-OST0000: new disk, initializing [ 5501.487986] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5501.525847] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5503.587656] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 5507.211534] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5507.599928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5507.603054] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5507.648406] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5508.836672] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5508.907526] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5510.233393] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5510.313687] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5512.677134] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5512.677611] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5512.678598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5512.680254] LustreError: Skipped 1 previous similar message [ 5512.686128] Lustre: Skipped 1 previous similar message [ 5517.043138] LustreError: 397357:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5517.127559] Lustre: server umount lustre-OST0000 complete [ 5518.817525] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5518.822674] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5518.834343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5518.837616] Lustre: Skipped 1 previous similar message [ 5522.913109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5522.913592] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5524.612513] LustreError: 397558:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5524.615259] LustreError: 397558:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5524.644608] Lustre: server umount lustre-MDT0000 complete [ 5526.140349] LustreError: 394409:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759818436 with bad export cookie 8630647913472296406 [ 5526.145182] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5526.147526] LustreError: 394409:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5526.226886] LustreError: 397759:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5526.230269] LustreError: 397759:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5526.329333] Lustre: server umount lustre-MDT0001 complete [ 5529.380623] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 02:27:19 (1759818439) [ 5533.045297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5536.570210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5536.804335] Lustre: lustre-MDT0000: new disk, initializing [ 5536.845660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5538.202453] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 5541.253540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5541.380682] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5542.649454] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 5544.519717] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5546.109735] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5551.074881] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5551.075649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5551.080280] Lustre: Skipped 1 previous similar message [ 5551.084601] Lustre: Skipped 2 previous similar messages [ 5556.193208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5556.194948] Lustre: Skipped 1 previous similar message [ 5561.312109] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5561.347543] LustreError: 400366:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5561.350210] LustreError: 400366:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5561.385822] Lustre: server umount lustre-MDT0000 complete [ 5562.887839] LustreError: 400369:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759818472 with bad export cookie 8630647913472297274 [ 5562.893402] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5563.049593] Lustre: server umount lustre-MDT0001 complete [ 5566.349161] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 02:27:56 (1759818476) [ 5568.521805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5568.691690] LustreError: 401291:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5568.698598] LustreError: 401291:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5568.718160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5570.034663] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 5573.049479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5574.428685] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 5576.186824] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5577.428319] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5579.814952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5582.080938] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 5584.484985] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5597.847917] Lustre: Permanently deactivating lustre-OST0000 [ 5597.854718] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5605.360800] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5606.729555] LustreError: 404054:0:(mgs_llog.c:1937:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 5606.740838] LustreError: 404054:0:(mgs_llog.c:1991:mgs_replace_nids()) Only MGS is allowed to be started [ 5606.745833] LustreError: 404054:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5607.377277] LustreError: 404102:0:(mgs_llog.c:1991:mgs_replace_nids()) Only MGS is allowed to be started [ 5607.380096] LustreError: 404102:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5614.063123] LustreError: 404199:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5614.065277] LustreError: 404199:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5614.138235] Lustre: server umount lustre-OST0000 complete [ 5615.103869] LustreError: 404353:0:(mgs_llog.c:1991:mgs_replace_nids()) Only MGS is allowed to be started [ 5615.107460] LustreError: 404353:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5619.168822] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5619.170965] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5619.174565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5619.176082] Lustre: Skipped 2 previous similar messages [ 5625.318720] LustreError: 404450:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5625.322176] LustreError: 404450:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5625.354185] Lustre: server umount lustre-MDT0000 complete [ 5626.585070] LustreError: 402910:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759818536 with bad export cookie 8630647913472297806 [ 5626.588599] LustreError: 402910:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 5626.588814] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5636.153198] Lustre: server umount lustre-MDT0001 complete [ 5638.996492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5640.278415] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 5641.258426] LustreError: 405492:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 5643.603741] Lustre: 405831:0:(mgs_llog.c:1698:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.153@tcp [ 5644.393267] LustreError: 405928:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5644.395615] LustreError: 405928:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5644.426090] Lustre: server umount MGS complete [ 5648.846932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5648.999244] LustreError: 406555:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5649.006350] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5649.008767] Lustre: Skipped 1 previous similar message [ 5649.033203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5649.036442] Lustre: Skipped 2 previous similar messages [ 5650.404409] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 5653.996788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5654.097785] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5655.410872] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 5657.209528] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5658.366798] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5659.020513] Lustre: Permanently reactivating lustre-OST0000 [ 5659.026704] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5659.029566] Lustre: Skipped 2 previous similar messages [ 5685.216142] LustreError: 408159:0:(osp_dev.c:692:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 5688.753677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5688.874258] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5688.876365] Lustre: Skipped 1 previous similar message [ 5690.769528] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 5692.520212] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5705.191271] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5705.193732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5705.199156] Lustre: Skipped 2 previous similar messages [ 5705.199592] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5705.202747] Lustre: Skipped 5 previous similar messages [ 5706.606312] LustreError: 409938:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5706.608735] LustreError: 409938:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5706.678571] Lustre: server umount lustre-OST0000 complete [ 5710.064257] Lustre: server umount lustre-MDT0000 complete [ 5710.307658] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5710.309327] LustreError: 406554:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5710.319171] LustreError: 406554:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5711.689108] LustreError: 406537:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759818621 with bad export cookie 8630647913472298758 [ 5711.693755] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5711.693983] LustreError: 406537:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5711.869868] Lustre: server umount lustre-MDT0001 complete [ 5715.670909] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 5716.757967] Key type lgssc unregistered [ 5716.889393] LNet: 410875:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5716.893847] LNetError: 410875:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5716.906426] LNet: Removed LNI 192.168.203.153@tcp [ 5717.228399] Key type .llcrypt unregistered [ 5717.229445] Key type ._llcrypt unregistered [ 5723.323133] Key type ._llcrypt registered [ 5723.324067] Key type .llcrypt registered [ 5723.364226] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 5729.665447] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 5729.969375] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5730.005054] alg: No test for adler32 (adler32-zlib) [ 5730.842239] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 5730.918219] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 5732.496150] Key type lgssc registered [ 5732.871369] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5735.722803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5737.982772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5740.317248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5743.161814] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5745.957420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5745.976100] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5745.981253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5747.060207] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5747.070938] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5747.106736] Lustre: lustre-MDT0000: new disk, initializing [ 5747.128925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5747.134788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5748.644060] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 5753.092389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5753.122882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5753.152927] Lustre: 413980:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5753.183364] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5753.186708] Lustre: Skipped 1 previous similar message [ 5753.227923] Lustre: lustre-MDT0001: new disk, initializing [ 5753.265872] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5753.279616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5753.282961] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5754.777879] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 5758.162571] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5759.481035] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5762.278899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5762.315718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5762.453441] Lustre: lustre-OST0000: new disk, initializing [ 5762.456453] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5762.498158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5764.570118] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 5767.803817] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5768.699782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5768.702651] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5768.737962] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5769.207712] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5769.282088] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5770.614900] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5770.691929] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5773.796282] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5773.796305] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5773.803947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5773.806194] Lustre: Skipped 1 previous similar message [ 5777.517102] LustreError: 416920:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5777.581220] Lustre: server umount lustre-OST0000 complete [ 5778.913667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5778.913687] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5778.914258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5778.916937] Lustre: Skipped 2 previous similar messages [ 5778.918611] LustreError: Skipped 1 previous similar message [ 5784.033112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5784.034794] Lustre: Skipped 2 previous similar messages [ 5789.153245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5789.155286] Lustre: Skipped 1 previous similar message [ 5793.248095] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5793.278266] LustreError: 417121:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5793.280980] LustreError: 417121:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5793.315200] Lustre: server umount lustre-MDT0000 complete [ 5794.570180] LustreError: 414761:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759818704 with bad export cookie 2750577649379013778 [ 5794.573875] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5794.578236] LustreError: 414761:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5794.641990] LustreError: 417324:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5794.645311] LustreError: 417324:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5794.725503] Lustre: server umount lustre-MDT0001 complete [ 5797.673913] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 5798.235181] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 02:31:48 (1759818708) [ 5800.512773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5800.660433] LustreError: 418042:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5800.665345] LustreError: 418042:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5800.684452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5801.950305] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 5804.739449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5804.868659] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5806.182399] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 5808.030373] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5809.300669] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5811.642466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5814.036377] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 5815.847689] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5816.677513] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 5827.657284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 5830.628069] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5830.630009] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5830.631437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5830.635079] Lustre: Skipped 1 previous similar message [ 5830.639910] Lustre: Skipped 1 previous similar message [ 5835.185250] LustreError: 420501:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5835.187750] LustreError: 420501:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5835.259125] Lustre: server umount lustre-OST0000 complete [ 5840.866094] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5840.866122] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5840.867048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5840.870194] LustreError: Skipped 1 previous similar message [ 5840.873722] Lustre: Skipped 1 previous similar message [ 5842.817370] LustreError: 420702:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5842.819025] LustreError: 420702:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5842.844374] Lustre: server umount lustre-MDT0000 complete [ 5844.091978] LustreError: 418067:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759818754 with bad export cookie 2750577649379014660 [ 5844.098691] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5844.099674] LustreError: 418067:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5844.253889] Lustre: server umount lustre-MDT0001 complete [ 5848.197749] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 5849.283275] Key type lgssc unregistered [ 5849.409326] LNet: 421433:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5849.411834] LNetError: 421433:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5849.419417] LNet: Removed LNI 192.168.203.153@tcp [ 5849.746076] Key type .llcrypt unregistered [ 5849.747367] Key type ._llcrypt unregistered [ 5857.125381] Key type ._llcrypt registered [ 5857.126370] Key type .llcrypt registered [ 5857.167951] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 5857.269308] Key type .llcrypt unregistered [ 5857.270518] Key type ._llcrypt unregistered [ 5863.969158] Key type ._llcrypt registered [ 5863.970279] Key type .llcrypt registered [ 5864.014750] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 5864.376537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5864.403189] alg: No test for adler32 (adler32-zlib) [ 5865.252751] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 5865.335849] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 5866.912126] Key type lgssc registered [ 5867.306864] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5870.041720] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5870.048254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5871.201143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5872.473357] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 5875.410047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5875.560072] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5877.502140] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 5879.272790] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5880.801819] LustreError: 423582:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5880.806661] LustreError: 423582:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5881.592173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5881.739515] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5882.800250] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5883.141943] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 5886.946324] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5886.949530] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5886.956362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5887.973839] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5887.974667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5887.982577] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5887.986238] Lustre: Skipped 1 previous similar message [ 5892.269759] LustreError: 425640:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 5892.338648] Lustre: server umount lustre-OST0000 complete [ 5897.184958] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5897.186549] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5897.187398] LustreError: Skipped 1 previous similar message [ 5897.193799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5899.709504] LustreError: 425841:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5899.712345] LustreError: 425841:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5899.744661] Lustre: server umount lustre-MDT0000 complete [ 5900.944057] LustreError: 423562:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759818811 with bad export cookie 7175606470283047304 [ 5900.948155] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5901.012831] LustreError: 426043:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5901.014490] LustreError: 426043:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5901.091281] Lustre: server umount lustre-MDT0001 complete [ 5904.637326] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 5905.815900] Key type lgssc unregistered [ 5905.946401] LNet: 426574:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5905.948374] LNetError: 426574:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5905.957330] LNet: Removed LNI 192.168.203.153@tcp [ 5906.253631] Key type .llcrypt unregistered [ 5906.255396] Key type ._llcrypt unregistered [ 5915.401069] Key type ._llcrypt registered [ 5915.402104] Key type .llcrypt registered [ 5915.446790] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 5915.993685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5916.015076] alg: No test for adler32 (adler32-zlib) [ 5916.909156] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 5917.008533] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 5918.616315] Key type lgssc registered [ 5919.197373] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5922.315939] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5922.326045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5937.824275] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5942.944354] LustreError: 427922:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.153@tcp: failed processing log, type 4: rc = -110 [ 5969.568203] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5971.746827] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 5974.013058] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5977.018541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5977.262116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5977.295097] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5978.829386] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 5982.454217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5982.622692] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5984.392877] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 5986.445323] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5988.008185] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5992.929702] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5992.930192] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5992.933558] LustreError: Skipped 1 previous similar message [ 5992.937579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5992.940040] Lustre: Skipped 1 previous similar message [ 5994.978763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5994.983193] Lustre: Skipped 1 previous similar message [ 5996.527353] LustreError: 430202:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 5996.591844] Lustre: server umount lustre-OST0000 complete [ 5998.048708] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5998.050832] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5998.053895] Lustre: Skipped 1 previous similar message [ 5998.055515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6000.097760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6000.099968] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6004.096667] LustreError: 430405:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 6004.099371] LustreError: 430405:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6004.131500] Lustre: server umount lustre-MDT0000 complete [ 6005.216920] LustreError: 428849:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6005.220863] LustreError: 428849:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6005.427218] LustreError: 430116:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759818915 with bad export cookie 190148631292856045 [ 6005.428918] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6005.429836] LustreError: 430116:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6005.494141] LustreError: 430609:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6005.495971] LustreError: 430609:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6005.577839] Lustre: server umount lustre-MDT0001 complete [ 6009.187823] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 6010.209040] Key type lgssc unregistered [ 6010.352414] LNet: 431140:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6010.354283] LNetError: 431140:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6010.360414] LNet: Removed LNI 192.168.203.153@tcp [ 6010.697896] Key type .llcrypt unregistered [ 6010.699526] Key type ._llcrypt unregistered [ 6020.568520] Key type ._llcrypt registered [ 6020.570136] Key type .llcrypt registered [ 6020.619661] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 6021.180706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6021.192883] alg: No test for adler32 (adler32-zlib) [ 6022.074691] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 6022.178136] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 6023.784333] Key type lgssc registered [ 6024.354699] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6027.709523] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6027.720145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6028.934670] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6030.280714] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6033.255706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6033.376547] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6034.706452] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6036.366461] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6037.463099] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6039.685036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6039.795807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6041.704906] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6043.583745] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6054.124821] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6064.097185] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6064.101740] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6064.108740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6064.610932] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6064.614981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6064.617370] Lustre: Skipped 2 previous similar messages [ 6066.050236] LustreError: 434783:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6066.075513] Lustre: server umount lustre-MDT0000 complete [ 6066.791202] LustreError: 434134:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.53@tcp arrived at 1759818976 with bad export cookie 11651062910933246206 [ 6066.796770] LustreError: 434134:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6067.403199] LustreError: 434134:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759818977 with bad export cookie 11651062910933245590 [ 6067.407962] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6069.728962] LustreError: 433238:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6069.729416] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6069.730244] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6069.730248] Lustre: Skipped 1 previous similar message [ 6069.730895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6069.736073] LustreError: 433238:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6072.736992] LustreError: 433238:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6072.737283] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6072.740685] LustreError: 433238:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6073.519124] LustreError: 434986:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6073.522218] LustreError: 434986:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6073.584650] Lustre: server umount lustre-OST0000 complete [ 6075.722369] LustreError: 435339:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6075.724734] LustreError: 435339:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6075.804425] Lustre: server umount lustre-MDT0001 complete [ 6079.138749] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 6080.119528] Key type lgssc unregistered [ 6080.246407] LNet: 435870:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6080.248595] LNetError: 435870:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6080.255342] LNet: Removed LNI 192.168.203.153@tcp [ 6080.548834] Key type .llcrypt unregistered [ 6080.549858] Key type ._llcrypt unregistered [ 6089.299090] Key type ._llcrypt registered [ 6089.300080] Key type .llcrypt registered [ 6089.336339] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 6089.657721] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6089.709159] alg: No test for adler32 (adler32-zlib) [ 6090.560842] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 6090.639424] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 6092.216169] Key type lgssc registered [ 6092.562089] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6095.089757] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6095.096156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6096.263064] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6097.531898] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6100.239534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6100.360966] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6101.585706] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6103.229436] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6104.325332] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6106.491144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6106.602729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6108.444590] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6110.093607] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6110.705298] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6120.931581] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6120.931847] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6120.937995] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6120.943421] Lustre: Skipped 1 previous similar message [ 6121.469699] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.53@tcp (stopping) [ 6123.137593] LustreError: 439488:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6123.157811] Lustre: server umount lustre-MDT0001 complete [ 6126.051070] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6126.051535] LustreError: 437243:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6126.052248] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6126.052254] Lustre: Skipped 1 previous similar message [ 6126.052977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6126.062544] LustreError: 437243:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6130.861916] LustreError: 439691:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6130.863808] LustreError: 439691:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6130.927954] Lustre: server umount lustre-OST0000 complete [ 6131.169371] LustreError: 437243:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6132.215926] LustreError: 439892:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6132.217979] LustreError: 439892:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6132.306626] Lustre: server umount lustre-MDT0000 complete [ 6136.674898] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 6137.818640] Key type lgssc unregistered [ 6137.960559] LNet: 440575:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6137.963502] LNetError: 440575:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6137.974434] LNet: Removed LNI 192.168.203.153@tcp [ 6138.299250] Key type .llcrypt unregistered [ 6138.300462] Key type ._llcrypt unregistered [ 6146.146975] Key type ._llcrypt registered [ 6146.148006] Key type .llcrypt registered [ 6146.183352] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 6151.985528] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 6152.306601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6152.346254] alg: No test for adler32 (adler32-zlib) [ 6153.200424] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 6153.287445] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 6154.864106] Key type lgssc registered [ 6155.206267] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6157.979080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6160.011074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6161.814264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6163.650177] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6166.647614] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 6169.680444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6171.960408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6171.978918] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6171.986446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6173.082268] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6173.093952] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6173.137917] Lustre: lustre-MDT0000: new disk, initializing [ 6173.161971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6173.170159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6174.422794] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6177.915468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6180.171293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6180.198310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6180.300321] Lustre: lustre-OST0000: new disk, initializing [ 6180.301987] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6180.303459] Lustre: Skipped 1 previous similar message [ 6180.326176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6180.577208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6180.580394] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6180.588755] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6182.211130] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6185.249291] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6188.430357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6190.615643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6190.640062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6190.670971] Lustre: 444519:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6190.697244] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6190.732417] Lustre: lustre-MDT0001: new disk, initializing [ 6190.772821] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6190.785638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 6190.788239] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 6192.037879] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6195.598376] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6195.669534] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6196.943394] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6197.018914] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6201.315771] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6201.316239] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6201.318229] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6201.324563] Lustre: Skipped 1 previous similar message [ 6202.877878] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.53@tcp (stopping) [ 6202.881140] Lustre: Skipped 1 previous similar message [ 6204.671464] LustreError: 447939:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6204.696609] Lustre: server umount lustre-MDT0001 complete [ 6211.555174] LustreError: 446125:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6211.559848] LustreError: 446125:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6216.674733] LustreError: 444555:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6216.678495] LustreError: 444555:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6221.793942] LustreError: 444531:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6221.794260] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6221.794568] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6221.805863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6223.023225] LustreError: 448213:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 6223.024965] LustreError: 448213:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6223.088583] Lustre: server umount lustre-OST0000 complete [ 6230.602118] LustreError: 448415:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 6230.606351] LustreError: 448415:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6230.706296] Lustre: server umount lustre-MDT0000 complete [ 6235.124258] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 6236.296751] Key type lgssc unregistered [ 6236.429372] LNet: 449098:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6236.431606] LNetError: 449098:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6236.440537] LNet: Removed LNI 192.168.203.153@tcp [ 6236.825909] Key type .llcrypt unregistered [ 6236.828605] Key type ._llcrypt unregistered [ 6249.657776] Key type ._llcrypt registered [ 6249.658757] Key type .llcrypt registered [ 6249.695247] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 6249.809667] Key type .llcrypt unregistered [ 6249.810984] Key type ._llcrypt unregistered [ 6256.264620] Key type ._llcrypt registered [ 6256.265479] Key type .llcrypt registered [ 6256.299734] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 6256.622298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6256.631049] alg: No test for adler32 (adler32-zlib) [ 6257.482212] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 6257.559475] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 6259.136121] Key type lgssc registered [ 6259.478805] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6262.096711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6264.238848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6266.315365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6268.865201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6268.887628] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6268.894121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6269.985798] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6269.997683] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6270.040960] Lustre: lustre-MDT0000: new disk, initializing [ 6270.068041] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6270.075896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6271.365329] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6276.051095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6276.087197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6276.140043] Lustre: 452417:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6276.192575] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6276.195962] Lustre: Skipped 1 previous similar message [ 6276.244420] Lustre: lustre-MDT0001: new disk, initializing [ 6276.304675] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6276.328933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6276.336134] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6278.176843] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6281.397670] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6282.500985] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6284.743264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6284.771497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6284.869548] Lustre: lustre-OST0000: new disk, initializing [ 6284.872531] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6284.901689] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6286.753510] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6289.757874] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6292.984573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6292.987193] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6293.028110] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6298.082551] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6298.084846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6298.086848] Lustre: Skipped 2 previous similar messages [ 6298.088406] Lustre: Skipped 1 previous similar message [ 6303.202634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6303.204375] Lustre: Skipped 2 previous similar messages [ 6303.293376] LustreError: 454861:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6303.319751] Lustre: server umount lustre-MDT0000 complete [ 6304.605419] LustreError: 453197:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759819214 with bad export cookie 6566361349316025117 [ 6304.608811] LustreError: 453197:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6304.609844] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6304.666829] LustreError: 455063:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6304.669302] LustreError: 455063:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6304.750264] Lustre: server umount lustre-MDT0001 complete [ 6316.212361] LustreError: 455263:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6316.214808] LustreError: 455263:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6316.296587] Lustre: server umount lustre-OST0000 complete [ 6319.689224] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 02:40:29 (1759819229) [ 6320.047013] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6322.384114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6322.568267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6323.834918] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6326.918446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6327.039843] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6328.320155] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6330.229486] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6331.716914] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6334.230850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6334.259908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6334.328038] Lustre: Found index 0 for lustre-OST0000, updating log [ 6336.183453] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6338.113993] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6347.691747] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 6352.870070] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6352.872805] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6352.877232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6352.883952] Lustre: Skipped 1 previous similar message [ 6352.890307] Lustre: Skipped 1 previous similar message [ 6356.146451] LustreError: 458373:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6356.151536] LustreError: 458373:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6356.237369] Lustre: server umount lustre-OST0000 complete [ 6357.529228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6359.010184] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6359.010839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6359.013632] Lustre: Skipped 1 previous similar message [ 6359.017009] Lustre: Skipped 3 previous similar messages [ 6364.413323] LustreError: 458627:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6364.415036] LustreError: 458627:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6364.439823] Lustre: server umount lustre-MDT0000 complete [ 6365.153136] LustreError: 458208:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6365.159672] LustreError: 458208:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6365.607115] LustreError: 456070:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759819275 with bad export cookie 6566361349316026496 [ 6365.609762] LustreError: 456070:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6365.610286] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6365.754773] Lustre: server umount lustre-MDT0001 complete [ 6368.361443] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 02:41:18 (1759819278) [ 6368.696144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6369.019457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6369.350743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6369.666397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6369.997337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6370.308268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6370.634330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6370.977733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6371.323009] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6371.679979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6372.065771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6372.465349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6372.818565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6373.251566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6373.661347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6374.113785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6374.517499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6374.897341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6375.309257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6375.712801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6376.082285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6376.445585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6376.821316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6377.148306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6377.567528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6377.951383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6378.462967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6378.892359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6379.352524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6379.752880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6380.190723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6380.704668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6381.185754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6381.649200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6382.121306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6382.567145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6382.951925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6383.373749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6383.841380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6384.197164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6384.543220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6384.928341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6385.260683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6385.591451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6385.900245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6386.202485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6386.514962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6386.842570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6387.217904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6387.668327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6388.044983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6390.713340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6390.877376] LustreError: 462208:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6390.903221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6390.905069] Lustre: Skipped 1 previous similar message [ 6392.146863] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6394.825723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6396.197110] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6398.282314] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6399.480168] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6402.158701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6402.204372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6402.292280] Lustre: Found index 0 for lustre-OST0000, updating log [ 6404.423813] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6406.112227] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6410.723622] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6410.725732] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6410.728752] LustreError: Skipped 1 previous similar message [ 6410.737167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6410.742373] Lustre: Skipped 2 previous similar messages [ 6412.972561] LustreError: 464521:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6412.974295] LustreError: 464521:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6413.030670] Lustre: server umount lustre-OST0000 complete [ 6419.361297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6419.363021] Lustre: Skipped 10 previous similar messages [ 6420.368521] Lustre: server umount lustre-MDT0000 complete [ 6420.385680] LustreError: 462920:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6420.390248] LustreError: 462920:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6421.558105] LustreError: 462187:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759819331 with bad export cookie 6566361349316027567 [ 6421.561257] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6421.561468] LustreError: 462187:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6421.700669] Lustre: server umount lustre-MDT0001 complete [ 6422.635144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6428.967376] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 6432.029295] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 6436.318770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6439.307073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6441.869610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6444.546522] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6447.114415] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 02:42:37 (1759819357) [ 6471.817799] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 6472.839963] Key type lgssc unregistered [ 6472.965452] LNet: 468508:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6472.967590] LNetError: 468508:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6599.975153] LNet: Removed LNI 192.168.203.153@tcp [ 6600.292316] Key type .llcrypt unregistered [ 6600.293410] Key type ._llcrypt unregistered [ 6606.969918] Key type ._llcrypt registered [ 6606.970875] Key type .llcrypt registered [ 6607.014743] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 6607.417917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6607.486746] alg: No test for adler32 (adler32-zlib) [ 6608.354482] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 6608.447444] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 6610.032142] Key type lgssc registered [ 6610.435920] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6611.921197] LNet: Added LNI 192.168.203.153@tcp100 [8/256/0/180] [ 6612.430972] Lustre: DEBUG MARKER: SKIP: conf-sanity test_73c unable to configure net [ 6618.384715] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 6619.572977] Key type lgssc unregistered [ 6619.741460] LNet: 470376:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6619.743625] LNetError: 470376:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6619.746558] LNet: Removed LNI 192.168.203.153@tcp [ 6619.753386] LNet: Removed LNI 192.168.203.153@tcp100 [ 6620.057582] Key type .llcrypt unregistered [ 6620.059476] Key type ._llcrypt unregistered [ 6627.647478] Key type ._llcrypt registered [ 6627.648458] Key type .llcrypt registered [ 6627.684247] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 6634.078305] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 6634.442886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6634.484110] alg: No test for adler32 (adler32-zlib) [ 6635.336940] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 6635.421966] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 6637.000156] Key type lgssc registered [ 6637.376843] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6640.310723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6642.648318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6644.634458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6646.668556] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6648.937180] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 02:45:59 (1759819559) [ 6654.806214] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 6655.856548] Key type lgssc unregistered [ 6655.993367] LNet: 474388:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6655.996557] LNetError: 474388:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6656.003287] LNet: Removed LNI 192.168.203.153@tcp [ 6656.322357] Key type .llcrypt unregistered [ 6656.323337] Key type ._llcrypt unregistered [ 6662.835073] Key type ._llcrypt registered [ 6662.836031] Key type .llcrypt registered [ 6662.870081] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 6663.252131] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6663.304940] alg: No test for adler32 (adler32-zlib) [ 6664.169102] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 6664.255412] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 6665.840241] Key type lgssc registered [ 6666.283241] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6669.575957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6669.594243] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6669.600843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6670.685430] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6670.696913] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6670.738394] Lustre: lustre-MDT0000: new disk, initializing [ 6670.764271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6670.772520] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6672.260575] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6676.550763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6676.589174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6676.638679] Lustre: 475560:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6676.668913] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6676.671481] Lustre: Skipped 1 previous similar message [ 6676.714238] Lustre: lustre-MDT0001: new disk, initializing [ 6676.751118] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6676.764724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6676.767254] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6678.097269] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6681.223093] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6682.466287] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6684.712517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6684.734611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6684.831531] Lustre: lustre-OST0000: new disk, initializing [ 6684.833272] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6684.863554] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6686.692126] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6689.815952] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6690.990495] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing lnet_if_list [ 6691.330163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6691.334702] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6691.375535] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6692.611115] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 6692.624662] LNet: Added LNI 192.168.203.153@tcp42 [8/256/0/180] [ 6701.539535] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6701.540023] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6701.540881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6701.543517] LustreError: Skipped 1 previous similar message [ 6701.546448] Lustre: Skipped 1 previous similar message [ 6705.004085] LustreError: 478937:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6705.064982] Lustre: server umount lustre-OST0000 complete [ 6706.657509] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6706.657954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6706.660952] Lustre: Skipped 1 previous similar message [ 6706.662351] Lustre: Skipped 1 previous similar message [ 6711.777195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6711.778838] Lustre: Skipped 2 previous similar messages [ 6712.380055] LustreError: 479142:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6712.382025] LustreError: 479142:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6712.410071] Lustre: server umount lustre-MDT0000 complete [ 6713.564465] LustreError: 475551:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759819623 with bad export cookie 1282097764677591222 [ 6713.567077] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6713.569665] LustreError: 475551:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6713.630950] LustreError: 479344:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6713.632637] LustreError: 479344:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6713.710783] Lustre: server umount lustre-MDT0001 complete [ 6717.144138] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 6718.300977] Key type lgssc unregistered [ 6718.472223] LNet: 479884:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6718.474404] LNetError: 479884:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6718.476574] LNet: Removed LNI 192.168.203.153@tcp [ 6718.484606] LNet: Removed LNI 192.168.203.153@tcp42 [ 6718.787823] Key type .llcrypt unregistered [ 6718.788915] Key type ._llcrypt unregistered [ 6726.153348] Key type ._llcrypt registered [ 6726.154515] Key type .llcrypt registered [ 6726.197771] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 6726.576318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6726.591078] alg: No test for adler32 (adler32-zlib) [ 6727.477216] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 6727.568300] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 6729.160150] Key type lgssc registered [ 6729.563262] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6732.178559] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6732.185630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6733.336057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6734.579144] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6737.332052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6737.457420] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6738.735265] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6740.618727] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6741.923973] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6744.290550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6744.419463] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6746.353812] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6748.193955] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6762.512929] loop0: detected capacity change from 0 to 204800000 [ 6762.572079] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6762.577590] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6762.583890] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6762.589216] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6762.610585] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6762.629107] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6762.642913] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6762.665829] blk_update_request: operation not supported error, dev loop0, sector 176912 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6762.691792] blk_update_request: operation not supported error, dev loop0, sector 185112 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6762.721873] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6764.600465] loop0: detected capacity change from 0 to 204800000 [ 6764.660051] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6764.712075] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6766.330733] loop0: detected capacity change from 0 to 204800000 [ 6768.144643] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6768.167030] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6768.209901] Lustre: fs15246-OST0000: new disk, initializing [ 6768.211861] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 6768.229299] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6770.056506] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6773.821790] loop1: detected capacity change from 0 to 204800000 [ 6775.651761] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6775.678145] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6775.709564] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 6775.730909] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 6775.768434] Lustre: fs15246-MDT0000: new disk, initializing [ 6775.814178] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6775.823251] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6775.862160] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 6776.928655] Lustre: 485499:0:(client.c:1627:after_reply()) @@@ resending request on EINPROGRESS req@000000007a589da9 x1845304635719808/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1759819703 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 6777.122707] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6781.985900] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6781.988767] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6781.993916] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6786.538435] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6786.542844] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 6790.397016] LustreError: 486622:0:(obd_class.h:479:obd_check_dev()) Device 30 not setup [ 6790.427843] Lustre: server umount fs15246-MDT0000 complete [ 6791.655317] LustreError: 483207:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6791.660019] LustreError: 483207:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6793.468085] LustreError: 487124:0:(obd_class.h:479:obd_check_dev()) Device 32 not setup [ 6793.469682] LustreError: 487124:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6793.483350] Lustre: server umount fs15246-OST0000 complete [ 6798.817172] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6798.819125] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6798.822491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6799.840793] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6799.842993] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6799.846556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6801.890312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6801.892770] Lustre: Skipped 1 previous similar message [ 6807.009624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6807.011114] Lustre: Skipped 1 previous similar message [ 6811.104118] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6811.123153] LustreError: 487625:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6811.125608] LustreError: 487625:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 6811.197555] Lustre: server umount lustre-OST0000 complete [ 6814.176939] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6814.179680] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6817.249287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6817.251287] Lustre: Skipped 2 previous similar messages [ 6818.557893] LustreError: 487828:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6818.560395] LustreError: 487828:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6818.586819] Lustre: server umount lustre-MDT0000 complete [ 6819.806100] LustreError: 481085:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759819729 with bad export cookie 4746390525011913425 [ 6819.808891] LustreError: 481085:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6819.809399] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6819.958635] Lustre: server umount lustre-MDT0001 complete [ 6823.439643] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 6824.594524] Key type lgssc unregistered [ 6824.741443] LNet: 488562:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6824.743447] LNetError: 488562:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6824.750404] LNet: Removed LNI 192.168.203.153@tcp [ 6825.080925] Key type .llcrypt unregistered [ 6825.081860] Key type ._llcrypt unregistered [ 6832.339173] Key type ._llcrypt registered [ 6832.340089] Key type .llcrypt registered [ 6832.377833] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 6832.688722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6832.736495] alg: No test for adler32 (adler32-zlib) [ 6833.599850] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 6833.684659] LNet: Added LNI 192.168.203.153@tcp [8/256/0/180] [ 6835.272169] Key type lgssc registered [ 6835.654391] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6838.167755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6839.760977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6841.514731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6843.137031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6848.734627] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 6852.011700] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 6855.810038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6858.037577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6860.583544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6862.890587] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6866.003746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6866.038561] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6866.047345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6867.154452] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6867.168451] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6867.213788] Lustre: lustre-MDT0000: new disk, initializing [ 6867.239758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6867.246519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6868.673601] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6873.494371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6873.536051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6873.599409] Lustre: 492963:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6873.640194] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6873.645748] Lustre: Skipped 1 previous similar message [ 6873.691153] Lustre: lustre-MDT0001: new disk, initializing [ 6873.730834] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6873.749725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6873.754740] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6875.163255] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6878.431755] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6879.674515] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6882.294910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6882.318679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6882.428476] Lustre: lustre-OST0000: new disk, initializing [ 6882.430300] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6882.463481] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6884.388899] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_default_debug -1 all [ 6887.445035] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6888.443671] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6888.446979] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6888.481143] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6888.865415] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6888.948638] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6890.452597] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6890.528821] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6893.539453] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6893.539637] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6893.540707] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6893.546398] Lustre: Skipped 1 previous similar message [ 6893.549085] LustreError: Skipped 1 previous similar message [ 6897.324145] LustreError: 495900:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6897.405542] Lustre: server umount lustre-OST0000 complete [ 6899.681922] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6899.684601] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6899.689445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6899.691147] Lustre: Skipped 1 previous similar message [ 6903.777100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6903.777715] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6904.975864] LustreError: 496101:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6904.979137] LustreError: 496101:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6905.017788] Lustre: server umount lustre-MDT0000 complete [ 6906.556832] LustreError: 492953:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759819816 with bad export cookie 16826888239529996703 [ 6906.561350] LustreError: MGC192.168.203.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6906.562535] LustreError: 492953:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6906.645494] LustreError: 496304:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6906.648336] LustreError: 496304:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6906.743157] Lustre: server umount lustre-MDT0001 complete [ 6915.468854] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing set_hostid [ 6918.673535] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing load_modules_local [ 6922.680724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6925.545573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6927.665661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6929.997312] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6930.564846] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6775 sec ========== 02:50:40 (1759819840) [ 6931.110790] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 02:50:41 (1759819841) === [ 6932.210501] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 02:50:42 (1759819842) === [ 6946.845181] Lustre: DEBUG MARKER: oleg353-server.virtnet: executing unload_modules_local [ 6947.926766] Key type lgssc unregistered [ 6948.059453] LNet: 501154:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6948.062338] LNetError: 501154:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6948.069293] LNet: Removed LNI 192.168.203.153@tcp [ 6948.387088] Key type .llcrypt unregistered [ 6948.387983] Key type ._llcrypt unregistered