[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 492628892 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002380] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.006797] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008013] pid_max: default: 32768 minimum: 301 [ 0.009122] LSM: Security Framework initializing [ 0.010037] Yama: becoming mindful. [ 0.011030] SELinux: Initializing. [ 0.012052] *** VALIDATE selinux *** [ 0.020683] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024796] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026049] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027114] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028115] *** VALIDATE tmpfs *** [ 0.030421] *** VALIDATE proc *** [ 0.031265] *** VALIDATE cgroup *** [ 0.032011] *** VALIDATE cgroup2 *** [ 0.033264] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034148] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036031] Spectre V2 : User space: Vulnerable [ 0.037010] Speculative Store Bypass: Vulnerable [ 0.040179] debug: unmapping init [mem 0xffffffff8f459000-0xffffffff8f460fff] [ 0.042162] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043598] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044022] ... version: 2 [ 0.045012] ... bit width: 48 [ 0.046011] ... generic registers: 4 [ 0.047012] ... value mask: 0000ffffffffffff [ 0.048015] ... max period: 00007fffffffffff [ 0.049013] ... fixed-purpose events: 3 [ 0.050013] ... event mask: 000000070000000f [ 0.052228] rcu: Hierarchical SRCU implementation. [ 0.054439] smp: Bringing up secondary CPUs ... [ 0.055628] x86: Booting SMP configuration: [ 0.056025] .... node #0, CPUs: #1 #2 #3 [ 0.059240] smp: Brought up 1 node, 4 CPUs [ 0.061016] smpboot: Max logical packages: 1 [ 0.062019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.274097] node 0 deferred pages initialised in 211ms [ 0.278016] devtmpfs: initialized [ 0.279236] x86/mm: Memory block size: 128MB [ 0.281478] gcov: version magic: 0x41383552 [ 0.284460] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.288123] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.290332] pinctrl core: initialized pinctrl subsystem [ 0.292160] [ 0.292724] ************************************************************* [ 0.295012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.296008] ** ** [ 0.298016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.300016] ** ** [ 0.302012] ** This means that this kernel is built to expose internal ** [ 0.305014] ** IOMMU data structures, which may compromise security on ** [ 0.307012] ** your system. ** [ 0.309012] ** ** [ 0.311014] ** If you see this message and you are not debugging the ** [ 0.313012] ** kernel, report this immediately to your vendor! ** [ 0.315012] ** ** [ 0.317014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.319015] ************************************************************* [ 0.321570] NET: Registered protocol family 16 [ 0.323393] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.326066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.329069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.332106] cpuidle: using governor menu [ 0.338384] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.340615] PCI: Using configuration type 1 for base access [ 0.343145] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.354382] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.357026] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.361262] cryptd: max_cpu_qlen set to 1000 [ 0.363854] ACPI: Added _OSI(Module Device) [ 0.366018] ACPI: Added _OSI(Processor Device) [ 0.368013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.369013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.373648] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.380360] ACPI: Interpreter enabled [ 0.382081] ACPI: PM: (supports S0 S3 S4 S5) [ 0.383041] ACPI: Using IOAPIC for interrupt routing [ 0.385136] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.388408] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.398367] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.401043] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.404020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.407080] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.411750] acpiphp: Slot [2] registered [ 0.413193] acpiphp: Slot [3] registered [ 0.415114] acpiphp: Slot [4] registered [ 0.416149] acpiphp: Slot [5] registered [ 0.417186] acpiphp: Slot [6] registered [ 0.419127] acpiphp: Slot [7] registered [ 0.420120] acpiphp: Slot [8] registered [ 0.422226] acpiphp: Slot [9] registered [ 0.424170] acpiphp: Slot [10] registered [ 0.426213] acpiphp: Slot [11] registered [ 0.428166] acpiphp: Slot [12] registered [ 0.429101] acpiphp: Slot [13] registered [ 0.431116] acpiphp: Slot [14] registered [ 0.432119] acpiphp: Slot [15] registered [ 0.434134] acpiphp: Slot [16] registered [ 0.435157] acpiphp: Slot [17] registered [ 0.437177] acpiphp: Slot [18] registered [ 0.438136] acpiphp: Slot [19] registered [ 0.440118] acpiphp: Slot [20] registered [ 0.441147] acpiphp: Slot [21] registered [ 0.443129] acpiphp: Slot [22] registered [ 0.445132] acpiphp: Slot [23] registered [ 0.447116] acpiphp: Slot [24] registered [ 0.448413] acpiphp: Slot [25] registered [ 0.450154] acpiphp: Slot [26] registered [ 0.452112] acpiphp: Slot [27] registered [ 0.456141] acpiphp: Slot [28] registered [ 0.458117] acpiphp: Slot [29] registered [ 0.459111] acpiphp: Slot [30] registered [ 0.461165] acpiphp: Slot [31] registered [ 0.463072] PCI host bridge to bus 0000:00 [ 0.464021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.466028] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.469029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.471026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.473025] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.476039] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.478256] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.482676] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.487731] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.497816] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.503584] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.507039] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.508020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.510020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.514119] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.516821] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.520053] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.523643] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.529020] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.542030] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.547872] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.553238] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.559025] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.565021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.583023] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.592314] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.597020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.606040] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.622018] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.632922] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.636077] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.643024] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.657025] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.667807] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.673019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.679019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.694029] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.701000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.705020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.713021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.729029] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.739368] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.743016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.748018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.762016] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.771000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.773360] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.775574] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.778386] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.782294] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.787044] iommu: Default domain type: Passthrough [ 0.789447] SCSI subsystem initialized [ 0.791147] ACPI: bus type USB registered [ 0.792143] usbcore: registered new interface driver usbfs [ 0.794069] usbcore: registered new interface driver hub [ 0.796078] usbcore: registered new device driver usb [ 0.797193] pps_core: LinuxPPS API ver. 1 registered [ 0.799013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.802110] PTP clock support registered [ 0.804180] EDAC MC: Ver: 3.0.0 [ 0.805489] PCI: Using ACPI for IRQ routing [ 0.807101] NetLabel: Initializing [ 0.808012] NetLabel: domain hash size = 128 [ 0.809426] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.812044] NetLabel: unlabeled traffic allowed by default [ 0.817086] vgaarb: loaded [ 0.818277] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.820019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.827615] clocksource: Switched to clocksource kvm-clock [ 0.934329] VFS: Disk quotas dquot_6.6.0 [ 0.935972] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.938980] *** VALIDATE ramfs *** [ 0.940495] *** VALIDATE hugetlbfs *** [ 0.942253] pnp: PnP ACPI init [ 0.945655] pnp: PnP ACPI: found 6 devices [ 0.971400] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.974505] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.977101] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.979643] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.982026] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.984726] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.987549] NET: Registered protocol family 2 [ 0.990138] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.994964] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.998446] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.003839] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.007256] TCP: Hash tables configured (established 65536 bind 65536) [ 1.010660] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.014129] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.017185] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.019562] NET: Registered protocol family 1 [ 1.022135] RPC: Registered named UNIX socket transport module. [ 1.024488] RPC: Registered udp transport module. [ 1.026582] RPC: Registered tcp transport module. [ 1.028351] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.030703] NET: Registered protocol family 44 [ 1.032474] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.034751] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.037123] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.039673] PCI: CLS 0 bytes, default 64 [ 1.042188] Unpacking initramfs... [ 2.483451] debug: unmapping init [mem 0xffff905ffcc54000-0xffff905ffffbffff] [ 2.487639] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.490355] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.493770] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.998128] Initialise system trusted keyrings [ 2.999932] Key type blacklist registered [ 3.001952] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.010823] zbud: loaded [ 3.013986] *** VALIDATE nfs *** [ 3.015444] *** VALIDATE nfs4 *** [ 3.017090] pstore: using deflate compression [ 3.019732] Platform Keyring initialized [ 3.120416] NET: Registered protocol family 38 [ 3.122674] Key type asymmetric registered [ 3.124380] Asymmetric key parser 'x509' registered [ 3.126294] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.129493] io scheduler mq-deadline registered [ 3.131301] io scheduler kyber registered [ 3.132881] io scheduler bfq registered [ 3.138242] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.141668] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.144809] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.148807] ACPI: Power Button [PWRF] [ 3.246488] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.338027] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.521820] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.619253] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.818184] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.850035] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.879697] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.885818] Non-volatile memory driver v1.3 [ 3.887439] Linux agpgart interface v0.103 [ 3.920331] virtio_blk virtio1: [vda] 133624 512-byte logical blocks (68.4 MB/65.2 MiB) [ 3.922797] vda: detected capacity change from 0 to 68415488 [ 3.935492] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.937890] vdb: detected capacity change from 0 to 1073741824 [ 3.953801] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.957582] vdc: detected capacity change from 0 to 2621440000 [ 3.971798] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.974990] vdd: detected capacity change from 0 to 2621440000 [ 3.989313] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.992409] vde: detected capacity change from 0 to 4294967296 [ 4.007772] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.010508] vdf: detected capacity change from 0 to 4294967296 [ 4.016818] libphy: Fixed MDIO Bus: probed [ 4.024787] usbcore: registered new interface driver usbserial_generic [ 4.027333] usbserial: USB Serial support registered for generic [ 4.029346] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.033449] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.035401] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.037665] mousedev: PS/2 mouse device common for all mice [ 4.041093] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.042879] rtc_cmos 00:05: RTC can wake from S4 [ 4.047264] rtc_cmos 00:05: registered as rtc0 [ 4.049957] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.051049] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.052817] intel_pstate: CPU model not supported [ 4.057493] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.062199] hid: raw HID events driver (C) Jiri Kosina [ 4.064838] usbcore: registered new interface driver usbhid [ 4.068634] usbhid: USB HID core driver [ 4.070323] drop_monitor: Initializing network drop monitor service [ 4.072952] Initializing XFRM netlink socket [ 4.075200] NET: Registered protocol family 10 [ 4.078342] Segment Routing with IPv6 [ 4.079530] NET: Registered protocol family 17 [ 4.081740] mpls_gso: MPLS GSO support [ 4.087148] RAS: Correctable Errors collector initialized. [ 4.089542] AVX version of gcm_enc/dec engaged. [ 4.091298] AES CTR mode by8 optimization enabled [ 4.173679] sched_clock: Marking stable (4173644496, 0)->(5101081115, -927436619) [ 4.177322] registered taskstats version 1 [ 4.180150] Loading compiled-in X.509 certificates [ 4.182466] zswap: loaded using pool lzo/zbud [ 4.207138] Key type big_key registered [ 4.228535] Key type encrypted registered [ 4.230187] ima: No TPM chip found, activating TPM-bypass! [ 4.232178] ima: Allocated hash algorithm: sha1 [ 4.233817] ima: No architecture policies found [ 4.235499] evm: Initialising EVM extended attributes: [ 4.237310] evm: security.selinux [ 4.238520] evm: security.ima [ 4.239617] evm: security.capability [ 4.240901] evm: HMAC attrs: 0x1 [ 4.243388] rtc_cmos 00:05: setting system clock to 2025-11-14 00:37:00 UTC (1763080620) [ 4.251368] debug: unmapping init [mem 0xffffffff90403000-0xffffffff905fffff] [ 4.253809] debug: unmapping init [mem 0xffffffff8f182000-0xffffffff8f458fff] [ 4.262119] Write protecting the kernel read-only data: 28672k [ 4.265312] debug: unmapping init [mem 0xffffffff8d803000-0xffffffff8d9fffff] [ 4.268026] debug: unmapping init [mem 0xffffffff8e114000-0xffffffff8e1fffff] [ 4.308582] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.316685] systemd[1]: Detected virtualization kvm. [ 4.318946] systemd[1]: Detected architecture x86-64. [ 4.320717] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.349223] systemd[1]: No hostname configured. [ 4.350946] systemd[1]: Set hostname to . [ 4.353216] random: systemd: uninitialized urandom read (16 bytes read) [ 4.355637] systemd[1]: Initializing machine ID from random generator. [ 4.403919] random: ln: uninitialized urandom read (6 bytes read) [ 4.497033] random: systemd: uninitialized urandom read (16 bytes read) [ 4.499541] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.504040] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 4.509142] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... Starting Setup Virtual Console... Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.181333] device-mapper: uevent: version 1.0.3 [ 5.183247] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ 5.867885] random: fast init done [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.931467] virtio_net virtio0 ens2: renamed from eth0 [ 5.956825] scsi host0: ata_piix [ 5.965675] scsi host1: ata_piix [ 5.967065] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.969111] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.420495] dracut-initqueue[594]: RTNETLINK answers: File exists [ 10.746264] random: crng init done [ 10.747518] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.239266] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Sockets. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.563540] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.812277] SELinux: Disabled at runtime. [ 12.878455] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.887215] systemd[1]: Detected virtualization kvm. [ 12.888867] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.360603] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.364206] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.369718] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.375445] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.379567] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.386708] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.391126] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Stopped target Switch Root. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 13.488367] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on Process Core Dump Socket. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.814467] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.147754] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.216248] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.342273] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.352500] EDAC sbridge: Ver: 1.1.2 [ 16.008476] Key type dns_resolver registered [ 16.316633] NFS: Registering the id_resolver key type [ 16.318782] Key type id_resolver registered [ 16.321322] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Crash recovery kernel arming. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg303-server login: [ 43.764502] libcfs: loading out-of-tree module taints kernel. [ 43.778088] Key type ._llcrypt registered [ 43.779716] Key type .llcrypt registered [ 43.836372] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 53.019874] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 54.770610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 54.812261] alg: No test for adler32 (adler32-zlib) [ 56.178755] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 56.966593] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 58.671389] Key type lgssc registered [ 59.990313] Lustre: Echo OBD driver; http://www.lustre.org/ [ 70.902017] hrtimer: interrupt took 2809877 ns [ 72.787970] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 73.932229] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 81.388143] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 87.523406] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 92.702354] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 104.100255] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 113.616845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 113.669520] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 113.684426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 114.932814] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 114.967780] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 115.041284] Lustre: lustre-MDT0000: new disk, initializing [ 115.117061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 115.140318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 118.146191] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 128.961169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 129.034946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 129.129677] Lustre: 6499:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 129.166592] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 129.171813] Lustre: Skipped 1 previous similar message [ 129.278594] Lustre: lustre-MDT0001: new disk, initializing [ 129.335195] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 129.359732] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 129.363918] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 132.725876] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 137.015192] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 145.101566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 145.166724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 145.346286] Lustre: lustre-OST0000: new disk, initializing [ 145.349927] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 145.428462] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 145.503454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 145.508623] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 145.590457] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 150.046943] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 161.442441] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 161.504041] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 161.585284] Lustre: lustre-OST0001: new disk, initializing [ 161.593980] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 161.643186] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 166.991222] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 168.004136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 168.015514] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 168.081934] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 177.147885] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 181.873711] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 191.257966] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing check_logdir /tmp/testlogs/ [ 194.717431] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing yml_node [ 198.890211] Lustre: DEBUG MARKER: Client: 2.16.61.1 [ 201.038137] Lustre: DEBUG MARKER: MDS: 2.16.61.1 [ 202.994900] Lustre: DEBUG MARKER: OSS: 2.16.61.1 [ 204.334496] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Nov 13 19:40:20 EST 2025 [ 218.023529] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 219.297594] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 234.464882] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 234.472714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 234.484764] Lustre: Skipped 1 previous similar message [ 234.510648] Lustre: Skipped 3 previous similar messages [ 239.586814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 239.589664] Lustre: Skipped 2 previous similar messages [ 240.477407] LustreError: 12205:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 240.632212] Lustre: server umount lustre-MDT0000 complete [ 244.145276] LustreError: 7443:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763080860 with bad export cookie 1088882744225869141 [ 244.149904] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 244.154582] LustreError: 7443:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 244.705876] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 244.711557] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 244.725831] Lustre: Skipped 3 previous similar messages [ 244.748981] Lustre: Skipped 2 previous similar messages [ 249.825945] LustreError: 8417:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 249.829908] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 249.840933] LustreError: 8417:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 249.863983] Lustre: Skipped 1 previous similar message [ 250.360810] LustreError: 12408:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 250.367410] LustreError: 12408:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 250.475904] Lustre: server umount lustre-MDT0001 complete [ 253.797154] LustreError: 12610:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 253.803667] LustreError: 12610:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 253.859481] Lustre: server umount lustre-OST0000 complete [ 257.061795] LustreError: 12810:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 257.072579] LustreError: 12810:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 257.203613] Lustre: server umount lustre-OST0001 complete [ 262.286889] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 269.507123] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 277.799690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 283.257373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 287.934474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 292.680683] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 298.310544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 298.388541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 298.644048] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 298.679664] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 298.770296] Lustre: lustre-MDT0000: new disk, initializing [ 298.859929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 298.883374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 302.339103] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 311.345394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 311.407423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 311.511722] Lustre: 14993:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 311.560842] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 311.563657] Lustre: Skipped 1 previous similar message [ 311.633169] Lustre: lustre-MDT0001: new disk, initializing [ 311.754588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 311.803838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 311.812408] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 315.440276] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 322.148269] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 325.474262] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 331.985191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 332.068366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 332.421727] Lustre: lustre-OST0000: new disk, initializing [ 332.442978] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 332.598623] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 333.922916] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 333.946961] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 334.133050] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 338.555920] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 345.454328] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 348.827761] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 349.013686] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 352.502907] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 352.685905] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 354.786957] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 354.787220] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 354.789857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 354.798866] Lustre: Skipped 1 previous similar message [ 354.802014] LustreError: Skipped 1 previous similar message [ 360.326164] LustreError: 17933:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 360.329032] LustreError: 17933:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 360.454561] Lustre: server umount lustre-OST0000 complete [ 365.032840] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 365.038554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 365.064206] Lustre: Skipped 1 previous similar message [ 365.071678] Lustre: Skipped 3 previous similar messages [ 378.336334] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 378.435642] LustreError: 18135:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 378.478686] LustreError: 18135:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 378.609128] Lustre: server umount lustre-MDT0000 complete [ 380.394769] LustreError: 15004:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 380.411400] LustreError: 15004:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 381.977851] LustreError: 14984:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763080998 with bad export cookie 1088882744225870674 [ 381.981169] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 382.019665] LustreError: 14984:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 382.497624] Lustre: server umount lustre-MDT0001 complete [ 386.082630] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 19:43:22 (1763081002) [ 388.289820] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 389.943521] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 19:43:26 (1763081006) [ 395.832766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 396.345372] LustreError: 19009:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 396.413591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 400.042548] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 401.890088] LustreError: 19009:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 405.987247] LustreError: 19009:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 407.651526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 411.404786] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 415.644217] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 418.710257] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 424.071439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 429.409369] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 434.126900] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 452.067266] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 452.068762] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 452.070703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 452.070710] Lustre: Skipped 5 previous similar messages [ 452.083885] Lustre: Skipped 1 previous similar message [ 456.439193] LustreError: 21276:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 456.462070] LustreError: 21276:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 456.610637] Lustre: server umount lustre-OST0000 complete [ 462.308848] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 462.313538] Lustre: Skipped 1 previous similar message [ 465.791960] Lustre: server umount lustre-MDT0000 complete [ 467.433116] LustreError: 20072:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 467.451716] LustreError: 20072:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 469.260675] LustreError: 18988:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763081085 with bad export cookie 1088882744225871556 [ 469.262586] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 469.278413] LustreError: 18988:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 469.709765] Lustre: server umount lustre-MDT0001 complete [ 480.045691] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 482.522327] Key type lgssc unregistered [ 482.765495] LNet: 22212:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 482.770722] LNetError: 22212:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 482.787600] LNet: Removed LNI 192.168.203.103@tcp [ 483.548557] Key type .llcrypt unregistered [ 483.551741] Key type ._llcrypt unregistered [ 503.119348] Key type ._llcrypt registered [ 503.120854] Key type .llcrypt registered [ 503.231221] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 504.138073] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 504.189100] alg: No test for adler32 (adler32-zlib) [ 505.177873] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 505.400923] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 507.063165] Key type lgssc registered [ 507.965879] Lustre: Echo OBD driver; http://www.lustre.org/ [ 514.836239] loop: module loaded [ 514.950624] loop0: detected capacity change from 0 to 204800000 [ 515.101351] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 515.121431] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 515.360710] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 515.444895] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 520.667180] loop0: detected capacity change from 0 to 204800000 [ 520.788533] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 520.794981] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 520.805775] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 520.895539] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 525.082584] loop0: detected capacity change from 0 to 204800000 [ 529.466473] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 529.510899] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 529.527395] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 530.774096] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 530.833839] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 530.918752] Lustre: lustre-MDT0000: new disk, initializing [ 531.019739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 531.047655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 535.039303] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 542.663113] loop1: detected capacity change from 0 to 204800000 [ 547.326298] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 547.370450] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 547.706506] Lustre: lustre-OST07c6: new disk, initializing [ 547.715400] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 547.722566] Lustre: Skipped 1 previous similar message [ 547.797351] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 553.248774] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 556.618346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 556.624721] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 556.662543] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 562.650810] Lustre: 25876:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 562.657499] Lustre: 25876:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 562.663655] Lustre: 25876:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 562.670250] Lustre: 25876:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 562.675282] Lustre: 25876:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 562.679279] Lustre: 25876:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 581.426137] Lustre: 24374:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 581.431076] Lustre: 24374:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 581.434969] Lustre: 24374:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 581.438829] Lustre: 24374:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 581.442284] Lustre: 24374:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 581.445061] Lustre: 24374:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 585.765233] Lustre: 24374:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 585.771490] Lustre: 24374:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 1 previous similar message [ 585.775461] Lustre: 24374:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 585.779411] Lustre: 24374:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 1 previous similar message [ 585.787292] Lustre: 24374:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 585.790885] Lustre: 24374:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 1 previous similar message [ 585.795903] Lustre: 24374:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 585.802800] Lustre: 24374:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 1 previous similar message [ 585.813119] Lustre: 24374:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 585.824383] Lustre: 24374:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 1 previous similar message [ 585.842046] Lustre: 24374:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 585.848466] Lustre: 24374:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 589.064478] Lustre: 24376:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 589.071107] Lustre: 24376:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 1 previous similar message [ 589.076490] Lustre: 24376:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 589.079964] Lustre: 24376:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 1 previous similar message [ 589.089255] Lustre: 24376:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 589.098191] Lustre: 24376:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 1 previous similar message [ 589.106179] Lustre: 24376:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 589.110059] Lustre: 24376:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 1 previous similar message [ 589.119123] Lustre: 24376:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 589.126357] Lustre: 24376:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 1 previous similar message [ 589.132516] Lustre: 24376:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 589.151449] Lustre: 24376:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 593.357722] Lustre: 24375:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 593.361298] Lustre: 24375:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 1 previous similar message [ 593.368154] Lustre: 24375:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 593.374640] Lustre: 24375:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 1 previous similar message [ 593.382229] Lustre: 24375:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 593.386591] Lustre: 24375:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 1 previous similar message [ 593.394845] Lustre: 24375:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 593.401821] Lustre: 24375:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 1 previous similar message [ 593.409078] Lustre: 24375:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 593.412632] Lustre: 24375:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 1 previous similar message [ 593.418623] Lustre: 24375:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 593.425533] Lustre: 24375:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 603.939324] Lustre: 24374:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 603.951143] Lustre: 24374:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 5 previous similar messages [ 603.955473] Lustre: 24374:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 603.963547] Lustre: 24374:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 603.968544] Lustre: 24374:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 603.972568] Lustre: 24374:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 603.977150] Lustre: 24374:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 603.982312] Lustre: 24374:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 603.990284] Lustre: 24374:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 603.993521] Lustre: 24374:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 603.997634] Lustre: 24374:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 604.001860] Lustre: 24374:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 653.794732] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 653.802107] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 653.811114] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 657.079101] LustreError: 26582:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 657.159020] Lustre: server umount lustre-OST07c6 complete [ 663.601646] LustreError: 27083:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 663.604409] LustreError: 27083:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 663.763434] Lustre: server umount lustre-MDT0000 complete [ 675.110223] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 677.007300] Key type lgssc unregistered [ 677.209192] LNet: 27917:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 677.214553] LNetError: 27917:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 677.228491] LNet: Removed LNI 192.168.203.103@tcp [ 677.769976] Key type .llcrypt unregistered [ 677.771711] Key type ._llcrypt unregistered [ 690.708859] Key type ._llcrypt registered [ 690.712949] Key type .llcrypt registered [ 690.797119] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 691.454505] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 691.522938] alg: No test for adler32 (adler32-zlib) [ 692.531684] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 692.713973] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 694.375151] Key type lgssc registered [ 695.256589] Lustre: Echo OBD driver; http://www.lustre.org/ [ 700.998379] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 701.028233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 702.447891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 706.109674] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 712.221624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 712.615439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 715.366106] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 720.402781] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 723.554279] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 730.370031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 730.779479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 736.200609] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 740.653794] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 748.207520] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 754.797181] LustreError: 31365:0:(qsd_reint.c:618:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:2 until pending updates are flushed. [ 762.849605] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 762.855075] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 762.872125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 763.876447] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 763.877700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 763.878633] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 763.896449] Lustre: Skipped 1 previous similar message [ 767.230197] LustreError: 31616:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 767.386287] Lustre: server umount lustre-OST0000 complete [ 774.114816] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 774.120412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 774.123638] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 774.132675] Lustre: Skipped 1 previous similar message [ 776.460818] LustreError: 31817:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 776.469513] LustreError: 31817:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 776.535784] Lustre: server umount lustre-MDT0000 complete [ 779.100537] LustreError: 29127:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763081395 with bad export cookie 15581214788840655650 [ 779.101168] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 779.104986] LustreError: 29127:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 779.211206] LustreError: 32020:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 779.214902] LustreError: 32020:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 779.438499] Lustre: server umount lustre-MDT0001 complete [ 787.023754] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 789.114803] Key type lgssc unregistered [ 789.317914] LNet: 32552:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 789.323912] LNetError: 32552:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 789.339595] LNet: Removed LNI 192.168.203.103@tcp [ 789.868814] Key type .llcrypt unregistered [ 789.874148] Key type ._llcrypt unregistered [ 804.587877] Key type ._llcrypt registered [ 804.595428] Key type .llcrypt registered [ 804.724595] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 805.706860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 805.853747] alg: No test for adler32 (adler32-zlib) [ 807.100373] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 807.297588] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 808.967155] Key type lgssc registered [ 810.042437] Lustre: Echo OBD driver; http://www.lustre.org/ [ 816.745822] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 816.763660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 818.428497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 822.030274] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 827.875628] LustreError: 33782:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 827.881753] LustreError: 33782:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 828.137319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 828.391603] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 831.422630] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 835.221125] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 838.466021] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 844.459808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 844.870826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 850.329131] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 854.937516] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 864.310960] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 873.441764] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 873.455996] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 873.477760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 874.467139] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 874.469906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 874.472937] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 879.098977] LustreError: 36050:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 879.211720] Lustre: server umount lustre-OST0000 complete [ 884.705744] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 884.706320] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 884.723366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 884.736425] Lustre: Skipped 2 previous similar messages [ 888.523230] LustreError: 36253:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 888.611363] LustreError: 36253:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 888.743827] Lustre: server umount lustre-MDT0000 complete [ 889.824176] LustreError: 33783:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 889.835434] LustreError: 33783:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 891.957408] LustreError: 36051:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763081508 with bad export cookie 8542458828256432124 [ 891.962513] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 892.136222] LustreError: 36454:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 892.144310] LustreError: 36454:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 892.319402] Lustre: server umount lustre-MDT0001 complete [ 899.263800] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 900.935718] Key type lgssc unregistered [ 901.154034] LNet: 36985:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 901.160109] LNetError: 36985:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 901.181735] LNet: Removed LNI 192.168.203.103@tcp [ 901.772590] Key type .llcrypt unregistered [ 901.774249] Key type ._llcrypt unregistered [ 917.529365] Key type ._llcrypt registered [ 917.530965] Key type .llcrypt registered [ 917.597781] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 918.172925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 918.245862] alg: No test for adler32 (adler32-zlib) [ 919.243775] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 919.404922] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 921.047173] Key type lgssc registered [ 921.862745] Lustre: Echo OBD driver; http://www.lustre.org/ [ 926.730376] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 926.745413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 927.997730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 929.910751] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 935.424084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 935.688554] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 938.668490] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 942.988377] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 945.543091] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 950.267803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 950.508425] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 953.585137] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 955.127771] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 958.798648] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 966.624540] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 966.628380] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 966.639554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 970.421062] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.3@tcp (stopping) [ 971.238696] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 971.244167] Lustre: Skipped 1 previous similar message [ 972.698239] LustreError: 40609:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 972.773172] Lustre: server umount lustre-MDT0000 complete [ 975.473754] LustreError: 38344:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763081591 with bad export cookie 17949878896939132973 [ 975.486333] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 975.524986] LustreError: 38359:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 975.537304] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.3@tcp (stopping) [ 975.545107] LustreError: 38359:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 975.552734] Lustre: Skipped 3 previous similar messages [ 976.357576] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 976.361339] LustreError: 38359:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 980.384446] LustreError: 38359:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 980.386264] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 980.392625] Lustre: Skipped 1 previous similar message [ 981.640658] LustreError: 40813:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 981.642529] LustreError: 40813:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 981.749525] Lustre: server umount lustre-MDT0001 complete [ 1000.631792] LustreError: 41036:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1000.634977] LustreError: 41036:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1000.736646] Lustre: server umount lustre-OST0000 complete [ 1011.338811] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 1013.457823] Key type lgssc unregistered [ 1013.672665] LNet: 41874:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1013.680741] LNetError: 41874:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1013.691519] LNet: Removed LNI 192.168.203.103@tcp [ 1014.280527] Key type .llcrypt unregistered [ 1014.282243] Key type ._llcrypt unregistered [ 1029.868712] Key type ._llcrypt registered [ 1029.870291] Key type .llcrypt registered [ 1029.961160] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 1030.663431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1030.749561] alg: No test for adler32 (adler32-zlib) [ 1031.702230] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 1031.851349] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 1033.475193] Key type lgssc registered [ 1034.463328] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1040.707664] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1040.740588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1042.206173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1045.219644] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1051.954562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1052.201058] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1055.450142] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1060.512516] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1063.555588] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1068.942136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1069.200638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1073.782966] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1077.994551] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1088.682740] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1094.115239] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1094.118700] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1094.120307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1094.136802] Lustre: Skipped 1 previous similar message [ 1094.144979] Lustre: Skipped 1 previous similar message [ 1094.840746] Lustre: lustre-OST0000: Not available for connect from 192.168.203.3@tcp (stopping) [ 1099.233106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1099.246687] Lustre: Skipped 1 previous similar message [ 1099.513896] LustreError: 45515:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1099.636037] Lustre: server umount lustre-OST0000 complete [ 1118.688404] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1118.700776] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1118.723390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1119.715332] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1122.354334] LustreError: 45894:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1122.357987] LustreError: 45894:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1122.441227] Lustre: server umount lustre-MDT0000 complete [ 1123.815459] LustreError: 45413:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1123.823926] LustreError: 45413:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1125.766535] LustreError: 43232:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763081742 with bad export cookie 16031039664937968477 [ 1125.775716] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1140.191129] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1140.297207] LustreError: 46096:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1140.304480] LustreError: 46096:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1140.518692] Lustre: server umount lustre-MDT0001 complete [ 1151.247451] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 1154.605822] Key type lgssc unregistered [ 1154.857914] LNet: 46630:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1154.869824] LNetError: 46630:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1154.894833] LNet: Removed LNI 192.168.203.103@tcp [ 1155.968827] Key type .llcrypt unregistered [ 1155.981598] Key type ._llcrypt unregistered [ 1179.515451] Key type ._llcrypt registered [ 1179.517342] Key type .llcrypt registered [ 1179.616951] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 1180.494276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1180.526756] alg: No test for adler32 (adler32-zlib) [ 1181.533757] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 1181.788486] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 1183.463188] Key type lgssc registered [ 1184.575831] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1190.818590] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1190.835474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1192.200823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1195.632633] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1202.657506] LustreError: 48028:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1202.670311] LustreError: 48028:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1203.404315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1203.813559] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1206.749177] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1211.341566] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1214.633328] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1220.124392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1220.382849] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1225.052472] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1229.574175] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1236.783434] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1241.675599] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1245.199524] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1257.543940] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2025-11-1319h57m52s [ 1258.953119] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1262.576764] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1262.583825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1263.304689] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.3@tcp (stopping) [ 1263.330433] Lustre: Skipped 2 previous similar messages [ 1264.367296] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.3@tcp (stopping) [ 1264.371514] Lustre: Skipped 3 previous similar messages [ 1269.434549] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.3@tcp (stopping) [ 1275.359101] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1275.456573] LustreError: 50611:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1275.533455] Lustre: server umount lustre-MDT0000 complete [ 1275.573401] LustreError: 49238:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1278.275290] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1281.701217] LustreError: 49238:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1281.725708] LustreError: 49238:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1283.789361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1283.870248] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1284.119681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1284.206345] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1285.161068] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1287.150363] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1290.184852] Lustre: DEBUG MARKER: Wait for df (35993) ... [ 1291.542349] Lustre: DEBUG MARKER: done [ 1295.338552] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1295.339260] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1295.346181] Lustre: Skipped 2 previous similar messages [ 1295.351242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1295.353257] Lustre: Skipped 3 previous similar messages [ 1300.721498] LustreError: 51671:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1300.724870] LustreError: 51671:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1300.943785] Lustre: server umount lustre-OST0000 complete [ 1305.568597] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1305.577705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1305.613978] Lustre: Skipped 3 previous similar messages [ 1305.615408] Lustre: Skipped 2 previous similar messages [ 1310.775550] LustreError: 51873:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1310.778447] LustreError: 51873:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1310.889764] Lustre: server umount lustre-MDT0000 complete [ 1311.714179] LustreError: 48027:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1311.751200] LustreError: 48027:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1314.482135] LustreError: 48007:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763081930 with bad export cookie 12596718734514477186 [ 1314.486551] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1314.494329] LustreError: 48007:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1314.648145] LustreError: 52074:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1314.683726] LustreError: 52074:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1314.984456] Lustre: server umount lustre-MDT0001 complete [ 1323.627776] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 1325.671066] Key type lgssc unregistered [ 1325.876944] LNet: 52606:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1325.885935] LNetError: 52606:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1325.902229] LNet: Removed LNI 192.168.203.103@tcp [ 1326.569545] Key type .llcrypt unregistered [ 1326.570899] Key type ._llcrypt unregistered [ 1353.074985] Key type ._llcrypt registered [ 1353.076554] Key type .llcrypt registered [ 1353.132726] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 1363.876353] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 1364.637586] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1364.687756] alg: No test for adler32 (adler32-zlib) [ 1365.677483] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 1365.875508] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 1367.511189] Key type lgssc registered [ 1368.401768] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1375.013194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1379.843416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1384.875151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1389.777972] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1395.192991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1395.234897] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1395.247703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1396.470915] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1396.513464] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1396.593248] Lustre: lustre-MDT0000: new disk, initializing [ 1396.682543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1396.703716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1400.046684] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1408.374722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1408.436681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1408.509676] Lustre: 56532:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1408.575654] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1408.580799] Lustre: Skipped 1 previous similar message [ 1408.646885] Lustre: lustre-MDT0001: new disk, initializing [ 1408.737781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1408.761987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1408.766857] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1411.814485] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1417.922940] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1420.357600] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1425.235340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1425.300618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1425.505762] Lustre: lustre-OST0000: new disk, initializing [ 1425.510120] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1425.586385] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1430.472481] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1435.206144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1435.224786] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1435.313876] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1437.409895] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1440.930361] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1441.136709] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1444.251058] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1444.467996] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1450.978571] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1450.980655] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1450.986516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1451.008902] Lustre: Skipped 1 previous similar message [ 1452.214158] LustreError: 59482:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1452.343071] Lustre: server umount lustre-OST0000 complete [ 1456.097620] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1456.113120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1456.118712] Lustre: Skipped 2 previous similar messages [ 1461.216401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1461.219223] Lustre: Skipped 1 previous similar message [ 1461.470970] LustreError: 59685:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1461.475306] LustreError: 59685:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1461.589986] Lustre: server umount lustre-MDT0000 complete [ 1464.503671] LustreError: 56523:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763082080 with bad export cookie 2535392578222261553 [ 1464.508028] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1464.516069] LustreError: 56523:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1464.622632] LustreError: 59886:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1464.626456] LustreError: 59886:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1464.748453] Lustre: server umount lustre-MDT0001 complete [ 1471.242579] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 20:01:27 (1763082087) [ 1476.606624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1476.955455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1479.742975] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1482.209220] LustreError: 60556:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1482.217980] LustreError: 60556:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1486.191698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1486.592963] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1489.771527] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1494.059644] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1497.140705] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1502.683221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1502.918879] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1508.136770] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1512.189975] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1526.668486] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2025-11-1320h02m23s [ 1527.778782] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1533.273789] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1535.124664] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1535.153921] LustreError: 62513:0:(mdt_handler.c:2310:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1555.657205] Lustre: lustre-MDT0000: Client 443bd5cb-467f-4f59-8673-059ab21096aa (at 192.168.203.3@tcp) reconnecting [ 1575.159132] LustreError: 62513:0:(mdt_handler.c:2310:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1577.020911] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1579.044852] Lustre: DEBUG MARKER: done [ 1583.072217] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1583.076780] LustreError: Skipped 1 previous similar message [ 1583.079965] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1583.087497] Lustre: Skipped 1 previous similar message [ 1583.091737] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1584.101749] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1589.053349] LustreError: 63580:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1589.056104] LustreError: 63580:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1589.164810] Lustre: server umount lustre-OST0000 complete [ 1594.336779] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1594.337060] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1594.356834] Lustre: Skipped 1 previous similar message [ 1594.375997] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1594.380169] Lustre: Skipped 2 previous similar messages [ 1606.111163] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1606.176615] LustreError: 63783:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1606.184455] LustreError: 63783:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1606.262254] Lustre: server umount lustre-MDT0000 complete [ 1607.655028] LustreError: 60555:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1607.667752] LustreError: 60555:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1609.203434] LustreError: 60538:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763082225 with bad export cookie 2535392578222262435 [ 1609.210428] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1609.214979] LustreError: 60538:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1609.441415] Lustre: server umount lustre-MDT0001 complete [ 1617.118742] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 1619.448858] Key type lgssc unregistered [ 1619.724568] LNet: 64518:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1619.735656] LNetError: 64518:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1619.750912] LNet: Removed LNI 192.168.203.103@tcp [ 1620.475992] Key type .llcrypt unregistered [ 1620.478526] Key type ._llcrypt unregistered [ 1645.990790] Key type ._llcrypt registered [ 1645.992658] Key type .llcrypt registered [ 1646.151706] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 1664.175864] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 1666.075930] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1666.110508] alg: No test for adler32 (adler32-zlib) [ 1667.258480] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 1667.646906] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 1669.343874] Key type lgssc registered [ 1671.157349] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1681.099572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1686.626792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1692.499546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1697.750492] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1703.727479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1703.767605] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1703.784193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1705.129650] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1705.173219] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1705.261778] Lustre: lustre-MDT0000: new disk, initializing [ 1705.366922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1705.391253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1708.961088] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1718.637879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1718.702737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1718.776382] Lustre: 68467:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1718.851215] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1718.854436] Lustre: Skipped 1 previous similar message [ 1718.956489] Lustre: lustre-MDT0001: new disk, initializing [ 1719.088922] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1719.141628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1719.159981] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1722.107673] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1727.965454] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1730.817817] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1736.569899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1736.646753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1736.926741] Lustre: lustre-OST0000: new disk, initializing [ 1736.935437] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1737.064647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1742.670042] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1747.012314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1747.025589] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1747.153372] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1749.614287] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1753.555951] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1753.803464] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1757.886045] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1758.115310] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1762.786208] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1762.786865] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1762.806564] LustreError: Skipped 1 previous similar message [ 1762.810604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1762.834173] Lustre: Skipped 1 previous similar message [ 1762.876245] Lustre: Skipped 1 previous similar message [ 1765.829394] LustreError: 71403:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1765.979093] Lustre: server umount lustre-OST0000 complete [ 1770.465537] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1770.468517] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1770.475500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1773.024759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1773.030851] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1775.788462] LustreError: 71606:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1775.807283] LustreError: 71606:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1776.154816] Lustre: server umount lustre-MDT0000 complete [ 1779.744492] LustreError: 68457:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763082395 with bad export cookie 7695013090102559089 [ 1779.746167] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1779.761126] LustreError: 68457:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1779.883611] LustreError: 71808:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1779.901034] LustreError: 71808:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1780.167918] Lustre: server umount lustre-MDT0001 complete [ 1788.230586] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 20:06:44 (1763082404) [ 1789.790837] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1791.345457] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 20:06:47 (1763082407) [ 1800.520823] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 1803.723245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1804.083473] LustreError: 72857:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1804.105469] LustreError: 72857:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1804.151271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1805.112848] Lustre: Failing over lustre-MDT0000 [ 1805.125349] LustreError: 72885:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1805.139393] LustreError: 72885:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1805.156300] LustreError: 72885:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 1, retries 0, failed: rc = -5 [ 1805.299990] LustreError: 72936:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1805.312727] LustreError: 72936:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1805.436913] Lustre: server umount lustre-MDT0000 complete [ 1810.699533] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 20:07:07 (1763082427) [ 1816.288637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1816.594591] LustreError: 73505:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1816.656627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1819.832215] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1821.665708] LustreError: 73505:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1825.764820] LustreError: 73505:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1826.944881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1827.443221] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1830.783314] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1835.064555] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1837.496972] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1842.514784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1846.829257] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1850.608781] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1866.267339] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1871.705241] LustreError: 75871:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1871.713891] LustreError: 75871:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1871.868918] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1871.882887] LustreError: 75522:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1871.947439] Lustre: server umount lustre-MDT0000 complete [ 1875.139658] LustreError: 73486:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763082491 with bad export cookie 7695013090102560244 [ 1875.146435] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1875.418916] Lustre: server umount lustre-MDT0001 complete [ 1878.520642] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1882.552046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1889.411689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1893.279462] Lustre: 66677:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763082493/real 1763082493] req@000000001d7e0956 x1848725768048768/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763082509 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1893.320760] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1893.352706] Lustre: Skipped 2 previous similar messages [ 1897.441835] Lustre: 66677:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763082497/real 1763082497] req@0000000085e39231 x1848725768048896/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763082513 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1898.463287] Lustre: 66680:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763082498/real 1763082498] req@000000007bf1308f x1848725768049024/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763082514 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1900.513150] LustreError: 66676:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000085e39231 x1848725768049920/t0(0) o250->MGC192.168.203.103@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1900.713917] LustreError: 75115:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1900.726082] LustreError: 75115:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1900.805116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1900.813706] Lustre: Skipped 1 previous similar message [ 1900.917893] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1901.791633] Lustre: 66679:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763082501/real 1763082501] req@0000000070783acf x1848725768049280/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763082517 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1903.863115] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1905.952532] Lustre: 66677:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763082506/real 1763082506] req@00000000f79c6c1c x1848725768049408/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763082522 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1911.412322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1912.882746] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1915.768687] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1920.772133] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1924.410665] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1932.256217] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1932.261953] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1932.276915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1932.280279] Lustre: Skipped 1 previous similar message [ 1934.664937] LustreError: 78331:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1934.667521] LustreError: 78331:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1934.723155] Lustre: server umount lustre-MDT0000 complete [ 1937.066876] LustreError: 73484:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763082553 with bad export cookie 7695013090102562148 [ 1937.068139] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1937.072660] LustreError: 73484:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 1937.240253] Lustre: server umount lustre-MDT0001 complete [ 1941.523134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1949.237163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1953.760401] Lustre: 66680:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763082554/real 1763082554] req@0000000076083726 x1848725768073728/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763082570 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1953.784633] Lustre: 66680:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1953.790153] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1953.798115] Lustre: Skipped 2 previous similar messages [ 1962.024368] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x6aca2da2b18b52c0 [ 1962.041829] Lustre: MGC192.168.203.103@tcp: Connection restored to 0@lo (at 0@lo) [ 1962.044710] Lustre: Skipped 1 previous similar message [ 1962.381551] LustreError: 75114:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1962.390244] LustreError: 75114:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1962.457979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1962.466982] Lustre: Skipped 1 previous similar message [ 1962.513207] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1965.779642] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1972.634232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1973.923753] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1975.951226] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 1980.247938] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1983.026859] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1988.577287] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1988.595063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1988.621111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1988.633401] Lustre: Skipped 3 previous similar messages [ 1993.693563] LustreError: 80791:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1993.695632] LustreError: 80791:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1993.755484] Lustre: server umount lustre-MDT0000 complete [ 1994.721334] LustreError: 79391:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1994.732411] LustreError: 79391:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 1996.339929] LustreError: 73485:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763082612 with bad export cookie 7695013090102563520 [ 1996.346287] LustreError: 73485:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1996.346315] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1996.577488] Lustre: server umount lustre-MDT0001 complete [ 2001.553551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2004.352424] Lustre: DEBUG MARKER: files compared the same [ 2016.429087] Lustre: server umount lustre-OST0000 complete [ 2029.688098] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 2031.792731] Key type lgssc unregistered [ 2032.082569] LNet: 82425:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2032.091884] LNetError: 82425:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2032.110516] LNet: Removed LNI 192.168.203.103@tcp [ 2032.815645] Key type .llcrypt unregistered [ 2032.820096] Key type ._llcrypt unregistered [ 2049.506129] Key type ._llcrypt registered [ 2049.507786] Key type .llcrypt registered [ 2049.554848] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 2049.914671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2049.941505] alg: No test for adler32 (adler32-zlib) [ 2050.975193] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 2051.152474] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 2052.818357] Key type lgssc registered [ 2053.694776] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2060.650908] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2060.673938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2062.021405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2064.993082] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug +malloc all [ 2072.161057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2072.422293] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2075.189206] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug +malloc all [ 2079.455449] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2082.329760] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2087.874713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2088.129776] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2089.128151] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2092.742347] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug +malloc all [ 2097.387923] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2104.801875] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2104.819409] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2104.829913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2107.874290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2107.875881] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2107.889146] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2109.367143] LustreError: 86042:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2109.476572] Lustre: server umount lustre-OST0000 complete [ 2112.992440] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2113.001544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2113.011296] Lustre: Skipped 2 previous similar messages [ 2118.111769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2118.770316] LustreError: 86244:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2118.780782] LustreError: 86244:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2118.874980] Lustre: server umount lustre-MDT0000 complete [ 2121.886074] LustreError: 83780:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763082738 with bad export cookie 6026888804518372902 [ 2121.890409] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2121.905984] LustreError: 83780:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2122.024381] LustreError: 86446:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2122.028705] LustreError: 86446:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2122.175365] Lustre: server umount lustre-MDT0001 complete [ 2130.219509] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 2132.652207] Key type lgssc unregistered [ 2132.896092] LNet: 86977:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2132.907542] LNetError: 86977:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2132.925800] LNet: Removed LNI 192.168.203.103@tcp [ 2133.553448] Key type .llcrypt unregistered [ 2133.557737] Key type ._llcrypt unregistered [ 2151.466305] Key type ._llcrypt registered [ 2151.467948] Key type .llcrypt registered [ 2151.560445] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 2152.609606] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2152.645683] alg: No test for adler32 (adler32-zlib) [ 2153.715956] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 2153.951741] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 2155.631141] Key type lgssc registered [ 2156.766968] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2164.931733] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2164.965481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2180.512191] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2185.631275] LustreError: 88334:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.103@tcp: failed processing log, type 4: rc = -110 [ 2212.319499] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2217.375617] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2221.793911] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2228.613817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2228.766508] Lustre: *** cfs_fail_loc=706, val=0*** [ 2229.108580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2229.218108] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2233.063696] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2234.340316] LustreError: 88360:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2234.364240] LustreError: 88360:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2239.471429] LustreError: 88360:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2240.628357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2240.983625] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2244.211723] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2248.860641] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2251.847922] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2254.817691] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2254.820852] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2254.850981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2256.360446] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2256.363502] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2256.381577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2259.575140] LustreError: 90660:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 2259.702701] Lustre: server umount lustre-OST0000 complete [ 2266.597463] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2266.612525] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2266.640946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2266.671710] Lustre: Skipped 2 previous similar messages [ 2269.583594] LustreError: 90861:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2269.597209] LustreError: 90861:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2269.766455] Lustre: server umount lustre-MDT0000 complete [ 2272.748758] LustreError: 89308:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2272.761760] LustreError: 89308:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2272.950833] LustreError: 90862:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763082889 with bad export cookie 12598815621414936933 [ 2272.974556] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2273.097479] LustreError: 91064:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2273.115036] LustreError: 91064:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2273.290071] Lustre: server umount lustre-MDT0001 complete [ 2281.893720] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 2284.516960] Key type lgssc unregistered [ 2284.750728] LNet: 91595:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2284.756995] LNetError: 91595:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2284.768960] LNet: Removed LNI 192.168.203.103@tcp [ 2285.471503] Key type .llcrypt unregistered [ 2285.476528] Key type ._llcrypt unregistered [ 2304.302568] Key type ._llcrypt registered [ 2304.306292] Key type .llcrypt registered [ 2304.418981] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 2305.419684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2305.439092] alg: No test for adler32 (adler32-zlib) [ 2306.480869] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 2306.668715] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 2308.313218] Key type lgssc registered [ 2309.224211] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2315.575921] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2315.587661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2319.224113] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2325.006244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2325.407491] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2328.212804] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2330.591968] LustreError: 93642:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2330.612144] LustreError: 93642:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2334.636108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2334.925945] LustreError: 93642:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2334.933372] LustreError: 93642:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2334.954084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2335.973879] LustreError: 93645:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2335.986111] LustreError: 93645:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2338.538900] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2341.346966] LustreError: 93645:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2341.359530] LustreError: 93645:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2343.148130] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2343.154949] Lustre: MGS: Client fa72aeb4-ac55-4dfc-9bd9-b9cc7caf3593 (at 0@lo) reconnecting [ 2343.277492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2343.331572] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2345.123950] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2358.753852] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2358.758454] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2358.765361] Lustre: Skipped 1 previous similar message [ 2358.765745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2358.771155] Lustre: Skipped 1 previous similar message [ 2361.844253] LustreError: 95612:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2361.948589] Lustre: server umount lustre-OST0000 complete [ 2368.992427] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2368.995234] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2368.999492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2369.017606] Lustre: Skipped 1 previous similar message [ 2371.165703] LustreError: 95815:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2371.171876] LustreError: 95815:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2371.185360] Lustre: server umount lustre-MDT0000 complete [ 2374.112694] LustreError: 93640:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2374.127290] LustreError: 93640:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2374.757646] LustreError: 96016:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2374.762437] LustreError: 96016:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2374.869952] Lustre: server umount lustre-MDT0001 complete [ 2378.382452] LustreError: 96217:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2378.386023] LustreError: 96217:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2378.436700] Lustre: server umount MGS complete [ 2388.466668] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 20:16:44 (1763083004) [ 2412.611670] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 2420.489326] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 2429.078931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2434.627877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2439.423184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2444.885035] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2450.612529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2450.713663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2453.968917] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2462.297855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2462.379413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2462.594877] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2462.668160] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2462.749180] Lustre: lustre-MDT0001: new disk, initializing [ 2462.874366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2462.924880] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2462.948220] Lustre: 100736:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2462.951904] LustreError: 100736:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2466.715693] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2475.365481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2475.432403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2475.630449] Lustre: lustre-OST0000: new disk, initializing [ 2475.633390] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2475.702081] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2481.172388] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2484.838181] Lustre: 101929:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@000000002bd90028 x1848726438057344/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1763083117 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2487.926222] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2493.688078] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2493.695605] Lustre: MGS: Client 4e47f9b9-6586-4200-98b0-b37817d21e6f (at 0@lo) reconnecting [ 2493.732421] Lustre: 100019:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2493.746794] LustreError: 100019:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 2493.777514] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2493.846055] Lustre: lustre-MDT0000: new disk, initializing [ 2493.954243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2493.977605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2494.037171] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2495.072600] Lustre: 102449:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000ba8257e6 x1848726438063232/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1763083127 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2495.105137] Lustre: 102449:0:(client.c:1637:after_reply()) Skipped 1 previous similar message [ 2497.584302] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2500.216505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2500.226365] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2500.300443] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 2520.544212] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2520.549001] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2520.572752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2520.591365] Lustre: Skipped 1 previous similar message [ 2526.181197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2526.184123] Lustre: Skipped 2 previous similar messages [ 2531.300390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2531.305816] Lustre: Skipped 1 previous similar message [ 2534.367133] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2534.428151] LustreError: 102969:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 2534.444079] LustreError: 102969:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2534.639507] Lustre: server umount lustre-OST0000 complete [ 2541.537796] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2541.542862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2541.543164] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2541.555930] Lustre: Skipped 2 previous similar messages [ 2541.571313] Lustre: Skipped 2 previous similar messages [ 2552.287130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2552.375333] LustreError: 103173:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 2552.385945] LustreError: 103173:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2552.438506] Lustre: server umount lustre-MDT0000 complete [ 2570.211568] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2570.327719] LustreError: 103375:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 2570.334858] LustreError: 103375:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2570.499864] Lustre: server umount lustre-MDT0001 complete [ 2574.325433] Lustre: server umount MGS complete [ 2584.411143] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 20:20:00 (1763083200) [ 2601.050149] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 2603.593649] Key type lgssc unregistered [ 2603.905666] LNet: 104915:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2603.919359] LNetError: 104915:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2603.937756] LNet: Removed LNI 192.168.203.103@tcp [ 2604.686705] Key type .llcrypt unregistered [ 2604.689702] Key type ._llcrypt unregistered [ 2620.566620] Key type ._llcrypt registered [ 2620.569268] Key type .llcrypt registered [ 2620.666139] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 2621.819745] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2621.848865] alg: No test for adler32 (adler32-zlib) [ 2622.912608] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 2623.151769] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 2624.887285] Key type lgssc registered [ 2625.933505] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2633.030376] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2633.061854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2634.606563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2638.569236] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2643.950161] LustreError: 106116:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2643.973176] LustreError: 106116:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2646.102090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2646.561474] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2649.606501] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2653.245140] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2656.168315] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2662.107176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2662.485522] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2667.160116] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2671.410455] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2678.884629] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:33) [ 2688.480677] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2688.488581] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2688.506587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2689.512048] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2689.514661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2689.529990] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2694.480765] LustreError: 108382:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2694.560920] Lustre: server umount lustre-OST0000 complete [ 2697.698087] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2697.704696] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2697.728047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2697.730872] Lustre: Skipped 1 previous similar message [ 2699.744609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2699.748942] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2703.670777] LustreError: 108583:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2703.696067] LustreError: 108583:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2703.844595] Lustre: server umount lustre-MDT0000 complete [ 2704.866794] LustreError: 107188:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2704.890646] LustreError: 107188:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2706.804103] LustreError: 106098:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763083323 with bad export cookie 5497480637987022404 [ 2706.823073] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2706.948851] LustreError: 108785:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2706.951789] LustreError: 108785:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2707.138892] Lustre: server umount lustre-MDT0001 complete [ 2715.139194] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 2717.578067] Key type lgssc unregistered [ 2717.853349] LNet: 109316:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2717.862314] LNetError: 109316:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2718.889713] LNet: Removed LNI 192.168.203.103@tcp [ 2719.668834] Key type .llcrypt unregistered [ 2719.671627] Key type ._llcrypt unregistered [ 2731.021881] Key type ._llcrypt registered [ 2731.023670] Key type .llcrypt registered [ 2731.102388] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 2731.915239] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2732.028041] alg: No test for adler32 (adler32-zlib) [ 2733.054319] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 2733.245843] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 2734.935277] Key type lgssc registered [ 2735.973818] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2741.348736] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2741.351857] LustreError: 110360:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 2741.352954] Lustre: Skipped 1 previous similar message [ 2741.358179] LustreError: 110359:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 2741.363023] LustreError: 110360:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4993 [ 2741.365781] LustreError: 110360:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2741.372045] LustreError: 110360:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2741.390947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2742.908461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2749.364097] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2749.743071] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2753.242455] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2757.699224] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing lsmod [ 2760.175632] LustreError: 111528:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 2760.200774] LustreError: 111538:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 2760.203414] LustreError: 111528:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4985 [ 2760.205789] LustreError: 111528:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2760.214463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2760.216680] LustreError: 111528:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2760.589531] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2765.283281] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2765.288291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2765.296100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2765.801848] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2765.803774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2765.811855] Lustre: Skipped 1 previous similar message [ 2769.311573] LustreError: 111724:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2769.377319] Lustre: server umount lustre-MDT0000 complete [ 2770.916663] LustreError: 111030:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2770.924119] LustreError: 111030:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2772.041807] LustreError: 110380:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763083388 with bad export cookie 12870459508840874124 [ 2772.047027] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2772.051641] LustreError: 110380:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2772.162694] LustreError: 111929:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2772.164871] LustreError: 111929:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2772.306560] Lustre: server umount lustre-MDT0001 complete [ 2785.284085] LustreError: 112132:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2785.287596] LustreError: 112132:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2785.432760] Lustre: server umount lustre-OST0000 complete [ 2792.151149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2792.432051] LustreError: 112609:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2792.493808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2795.600518] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2797.540159] LustreError: 112609:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2801.545452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2801.789985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2804.446993] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2808.292652] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2811.039220] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2815.560967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2815.770164] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2817.838666] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:65) [ 2819.485950] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2823.220991] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2833.376218] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2833.383056] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2833.394211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2833.397931] Lustre: Skipped 1 previous similar message [ 2836.451118] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2836.453588] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2836.454099] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2836.475350] Lustre: Skipped 1 previous similar message [ 2838.584466] LustreError: 114854:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2838.590043] LustreError: 114854:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2838.670655] Lustre: server umount lustre-OST0000 complete [ 2841.569030] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2841.575476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2851.810287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2851.815315] Lustre: Skipped 4 previous similar messages [ 2855.394585] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2855.450602] LustreError: 115056:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2855.454199] LustreError: 115056:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2855.506631] Lustre: server umount lustre-MDT0000 complete [ 2855.910866] LustreError: 112608:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2855.921953] LustreError: 112608:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2857.805701] LustreError: 112588:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763083474 with bad export cookie 12870459508840874803 [ 2857.810320] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2857.822107] LustreError: 112588:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2858.086240] Lustre: server umount lustre-MDT0001 complete [ 2865.337615] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 2867.094208] Key type lgssc unregistered [ 2867.285595] LNet: 115790:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2867.293706] LNetError: 115790:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2867.306325] LNet: Removed LNI 192.168.203.103@tcp [ 2867.813584] Key type .llcrypt unregistered [ 2867.819520] Key type ._llcrypt unregistered [ 2882.860433] Key type ._llcrypt registered [ 2882.861946] Key type .llcrypt registered [ 2882.925535] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 2883.474277] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2883.569523] alg: No test for adler32 (adler32-zlib) [ 2884.519070] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 2884.682668] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 2886.295175] Key type lgssc registered [ 2887.047445] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2892.304816] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2892.328146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2893.653996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2896.081766] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2901.542747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2901.863884] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2904.773571] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2908.656132] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2910.906989] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2915.362793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2915.624408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2918.707298] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:67 to 0x240000400:97) [ 2919.702723] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2923.722962] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2931.215132] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2933.413141] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2937.312831] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2937.317215] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2937.326407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2937.330391] Lustre: Skipped 1 previous similar message [ 2937.906188] LustreError: 119509:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2938.041672] Lustre: server umount lustre-OST0000 complete [ 2942.434248] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2942.441266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2942.443932] Lustre: Skipped 1 previous similar message [ 2947.554812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2947.561938] Lustre: Skipped 1 previous similar message [ 2952.675073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2952.677675] Lustre: Skipped 1 previous similar message [ 2955.231153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2955.294909] LustreError: 119714:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2955.297542] LustreError: 119714:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2955.397471] Lustre: server umount lustre-MDT0000 complete [ 2956.772395] LustreError: 117950:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2956.790190] LustreError: 117950:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2957.874509] LustreError: 117147:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763083574 with bad export cookie 12513268664458468386 [ 2957.878058] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2957.881979] LustreError: 117147:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2957.973072] LustreError: 119917:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2957.976741] LustreError: 119917:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2958.096770] Lustre: server umount lustre-MDT0001 complete [ 2963.950517] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 2965.577210] Key type lgssc unregistered [ 2965.770505] LNet: 120448:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2965.778350] LNetError: 120448:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2966.823228] LNet: Removed LNI 192.168.203.103@tcp [ 2967.291276] Key type .llcrypt unregistered [ 2967.293814] Key type ._llcrypt unregistered [ 2977.082397] Key type ._llcrypt registered [ 2977.083872] Key type .llcrypt registered [ 2977.144252] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 2977.823303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2977.840958] alg: No test for adler32 (adler32-zlib) [ 2978.783670] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 2978.937245] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 2980.575602] Key type lgssc registered [ 2981.234978] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2985.737794] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2985.750448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2987.057764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2989.442882] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2994.432764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2994.632687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2996.719281] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 2999.425289] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3001.084942] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3004.274117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3004.457755] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3007.078230] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3009.723459] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3012.651694] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 3023.841429] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3023.848702] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3023.849905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3023.857362] Lustre: Skipped 1 previous similar message [ 3025.843127] LustreError: 123895:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3025.900639] Lustre: server umount lustre-OST0000 complete [ 3028.961050] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3028.972610] Lustre: Skipped 1 previous similar message [ 3028.976237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3028.982111] Lustre: Skipped 1 previous similar message [ 3034.080857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3034.083862] Lustre: Skipped 1 previous similar message [ 3039.201370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3042.271360] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3042.327760] LustreError: 124096:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3042.329686] LustreError: 124096:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3042.376438] Lustre: server umount lustre-MDT0000 complete [ 3044.323081] LustreError: 122341:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3044.336150] LustreError: 122341:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3044.666367] LustreError: 121609:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763083660 with bad export cookie 9500559616913297717 [ 3044.673099] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3044.678861] LustreError: 121609:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3044.793290] LustreError: 124298:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3044.796821] LustreError: 124298:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3044.939680] Lustre: server umount lustre-MDT0001 complete [ 3050.462712] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 3052.236973] Key type lgssc unregistered [ 3052.413160] LNet: 124829:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3052.416873] LNetError: 124829:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3052.429221] LNet: Removed LNI 192.168.203.103@tcp [ 3052.982825] Key type .llcrypt unregistered [ 3052.984678] Key type ._llcrypt unregistered [ 3068.166518] Key type ._llcrypt registered [ 3068.168158] Key type .llcrypt registered [ 3068.238203] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 3068.669706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3068.687729] alg: No test for adler32 (adler32-zlib) [ 3069.641115] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 3069.783479] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 3071.400641] Key type lgssc registered [ 3072.062903] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3076.406252] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3076.428476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3077.729504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3080.155369] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3085.304602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3085.507352] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3087.668914] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3090.509827] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3092.568622] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3096.222910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3096.405219] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3099.879990] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3102.570165] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:131 to 0x240000400:161) [ 3102.858690] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3117.261878] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3119.193325] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3119.196658] Lustre: Skipped 2 previous similar messages [ 3121.404605] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3121.408223] Lustre: Skipped 2 previous similar messages [ 3126.801506] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3126.820238] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3130.580399] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.3@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3130.586448] Lustre: Skipped 2 previous similar messages [ 3133.703561] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.3@tcp 192.168.0.[2,10]@tcp [ 3138.225126] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.3/24@tcp in log lustre-MDT0000 [ 3138.231574] Lustre: Skipped 2 previous similar messages [ 3142.345111] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.3/24@tcp [ 3142.347786] Lustre: Skipped 1 previous similar message [ 3146.721727] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3146.727866] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3146.732211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3146.735316] Lustre: Skipped 1 previous similar message [ 3151.542167] LustreError: 129922:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3151.612592] Lustre: server umount lustre-OST0000 complete [ 3156.962693] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3156.966878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3156.968660] Lustre: Skipped 1 previous similar message [ 3156.973360] Lustre: Skipped 2 previous similar messages [ 3162.081693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3162.084901] Lustre: Skipped 1 previous similar message [ 3167.201632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3168.223249] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3168.266046] LustreError: 130126:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3168.275177] LustreError: 130126:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3168.315171] Lustre: server umount lustre-MDT0000 complete [ 3170.295647] LustreError: 126185:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763083786 with bad export cookie 7633351254257504982 [ 3170.298123] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3170.301869] LustreError: 126185:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3170.406395] LustreError: 130328:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3170.408458] LustreError: 130328:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3170.503556] Lustre: server umount lustre-MDT0001 complete [ 3176.310228] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 3177.940386] Key type lgssc unregistered [ 3178.138033] LNet: 130859:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3178.142857] LNetError: 130859:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3178.157603] LNet: Removed LNI 192.168.203.103@tcp [ 3178.612707] Key type .llcrypt unregistered [ 3178.614954] Key type ._llcrypt unregistered [ 3193.509892] Key type ._llcrypt registered [ 3193.511539] Key type .llcrypt registered [ 3193.559518] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 3193.891246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3193.925380] alg: No test for adler32 (adler32-zlib) [ 3194.844175] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 3194.959985] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 3196.559162] Key type lgssc registered [ 3197.169430] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3200.831507] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3200.843740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3202.072443] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3202.096985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3202.099498] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.3/24@tcp [ 3203.930717] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3207.770883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3207.936520] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3207.948188] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.3/24@tcp [ 3207.981404] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3209.683228] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3212.071603] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3213.791083] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3217.269800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3217.458224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3220.258557] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3223.332244] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3227.688442] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:169 to 0x240000400:193) [ 3237.861696] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3237.861728] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3237.879922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3239.987280] LustreError: 134681:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3240.040560] Lustre: server umount lustre-OST0000 complete [ 3242.976798] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3242.979792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3242.981540] Lustre: Skipped 2 previous similar messages [ 3242.986260] Lustre: Skipped 2 previous similar messages [ 3248.096171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3248.099411] Lustre: Skipped 1 previous similar message [ 3253.218303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3253.221347] Lustre: Skipped 1 previous similar message [ 3256.288442] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3256.339457] LustreError: 134882:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3256.342800] LustreError: 134882:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3256.399253] Lustre: server umount lustre-MDT0000 complete [ 3258.337909] LustreError: 134595:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3258.348257] LustreError: 134595:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3258.512843] LustreError: 132359:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763083874 with bad export cookie 16873081355547014757 [ 3258.516741] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3258.520782] LustreError: 132359:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3258.603247] LustreError: 135084:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3258.607159] LustreError: 135084:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3258.708855] Lustre: server umount lustre-MDT0001 complete [ 3263.441621] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 3264.699673] Key type lgssc unregistered [ 3264.859996] LNet: 135615:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3264.867266] LNetError: 135615:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3264.877779] LNet: Removed LNI 192.168.203.103@tcp [ 3265.293558] Key type .llcrypt unregistered [ 3265.295107] Key type ._llcrypt unregistered [ 3275.850862] Key type ._llcrypt registered [ 3275.852877] Key type .llcrypt registered [ 3275.904721] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 3283.524506] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 3283.959362] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3284.006435] alg: No test for adler32 (adler32-zlib) [ 3284.896292] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 3284.999956] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 3286.599139] Key type lgssc registered [ 3287.085853] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3291.215551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3294.130985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3296.901363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3299.961197] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3303.330922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3303.361522] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3303.370756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3304.481227] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3304.501359] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3304.546388] Lustre: lustre-MDT0000: new disk, initializing [ 3304.595816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3304.607874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3306.398229] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3312.163312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3312.192142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3312.250328] Lustre: 138966:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3312.292697] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3312.295335] Lustre: Skipped 1 previous similar message [ 3312.336375] Lustre: lustre-MDT0001: new disk, initializing [ 3312.394907] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3312.416919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3312.422900] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3314.217844] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3318.283078] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3319.908574] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3323.217245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3323.251579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3323.401379] Lustre: lustre-OST0000: new disk, initializing [ 3323.405133] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3323.452702] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3326.442363] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3329.078378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3329.089585] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3329.161548] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3331.093680] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3333.224594] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3333.372582] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3335.804840] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3335.943331] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3339.240488] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3339.243873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3339.250848] Lustre: Skipped 1 previous similar message [ 3343.216241] LustreError: 141901:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3343.293743] Lustre: server umount lustre-OST0000 complete [ 3348.450385] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3348.455444] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3348.461422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3348.469337] Lustre: Skipped 1 previous similar message [ 3349.472586] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3349.475763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3349.480811] Lustre: Skipped 1 previous similar message [ 3351.501364] LustreError: 142102:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3351.507562] LustreError: 142102:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3351.549275] Lustre: server umount lustre-MDT0000 complete [ 3353.550818] LustreError: 138957:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763083969 with bad export cookie 8160429567320397968 [ 3353.555751] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3353.563585] LustreError: 138957:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3353.662046] LustreError: 142303:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3353.665129] LustreError: 142303:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3353.760928] Lustre: server umount lustre-MDT0001 complete [ 3358.958251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3359.253079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3361.550900] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3364.323194] LustreError: 142777:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3364.333991] LustreError: 142777:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3366.430399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3366.657780] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3368.779300] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3372.057075] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3374.383793] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3378.984646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3379.181897] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3382.366022] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3385.485720] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3387.701728] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3387.846748] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3390.171530] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3392.418522] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 2 sec [ 3397.743620] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3397.783479] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3397.905130] LustreError: 144376:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3397.916678] Lustre: lustre-OST0001: new disk, initializing [ 3397.924856] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3397.990532] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3401.012887] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3405.384256] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3405.857492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3405.863342] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3405.911634] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3406.537038] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3408.309577] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3408.422792] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3410.121834] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8903d12f1800.ost_server_uuid 50 [ 3410.801070] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8903d12f1800.ost_server_uuid in FULL state after 0 sec [ 3416.034241] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3416.037075] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3416.038857] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3416.042197] Lustre: Skipped 1 previous similar message [ 3420.085098] LustreError: 147042:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3420.087303] LustreError: 147042:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3420.126332] Lustre: server umount lustre-OST0001 complete [ 3421.160493] LustreError: 144375:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3421.167511] LustreError: 144375:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3422.177369] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3422.181011] LustreError: Skipped 1 previous similar message [ 3422.182944] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3422.188983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3422.192268] Lustre: Skipped 1 previous similar message [ 3426.277629] LustreError: 144375:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3426.278132] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3426.290094] LustreError: 144375:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3428.085208] LustreError: 147243:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3428.092226] LustreError: 147243:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3428.138429] Lustre: server umount lustre-OST0000 complete [ 3431.395423] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3431.401167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3431.410645] Lustre: Skipped 2 previous similar messages [ 3431.419485] Lustre: Skipped 3 previous similar messages [ 3436.246375] LustreError: 147446:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3436.248615] LustreError: 147446:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3436.301380] Lustre: server umount lustre-MDT0000 complete [ 3436.513214] LustreError: 143814:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3436.529053] LustreError: 143814:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3438.236382] LustreError: 142760:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763084054 with bad export cookie 8160429567320398850 [ 3438.241660] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3438.241940] LustreError: 142760:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3438.418975] Lustre: server umount lustre-MDT0001 complete [ 3447.580400] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 3448.922332] Key type lgssc unregistered [ 3449.071702] LNet: 148636:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3449.080343] LNetError: 148636:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3449.090981] LNet: Removed LNI 192.168.203.103@tcp [ 3449.516965] Key type .llcrypt unregistered [ 3449.519375] Key type ._llcrypt unregistered [ 3465.642128] Key type ._llcrypt registered [ 3465.643759] Key type .llcrypt registered [ 3465.695121] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 3473.488764] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 3474.020323] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3474.048705] alg: No test for adler32 (adler32-zlib) [ 3474.946570] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 3475.056648] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 3476.663336] Key type lgssc registered [ 3477.323597] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3482.122390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3485.036659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3487.863152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3491.419243] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3495.098630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3495.122810] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3495.130049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3496.239030] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3496.258902] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3496.300926] Lustre: lustre-MDT0000: new disk, initializing [ 3496.344873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3496.356558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3498.057527] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3503.051823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3503.076750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3503.126691] Lustre: 152557:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3503.161383] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3503.164271] Lustre: Skipped 1 previous similar message [ 3503.203830] Lustre: lustre-MDT0001: new disk, initializing [ 3503.246638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3503.265813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3503.271426] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3504.911953] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3508.579356] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3510.133293] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3512.991794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3513.024356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3513.138501] Lustre: lustre-OST0000: new disk, initializing [ 3513.141235] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3513.174758] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3515.649885] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3516.157185] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3516.163259] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3516.202370] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3519.187928] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3520.819812] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3520.915894] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3522.497327] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3522.600319] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3526.625698] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3526.626034] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3526.628657] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3526.628783] LustreError: Skipped 1 previous similar message [ 3526.638762] Lustre: Skipped 1 previous similar message [ 3529.453112] LustreError: 155498:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3529.504643] Lustre: server umount lustre-OST0000 complete [ 3531.746415] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3531.747118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3531.750924] Lustre: Skipped 1 previous similar message [ 3531.753857] Lustre: Skipped 1 previous similar message [ 3536.864758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3536.870537] Lustre: Skipped 2 previous similar messages [ 3537.032930] LustreError: 155699:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3537.035720] LustreError: 155699:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3537.072101] Lustre: server umount lustre-MDT0000 complete [ 3538.629314] LustreError: 153338:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763084154 with bad export cookie 4020181992264403429 [ 3538.632885] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3538.633704] LustreError: 153338:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3538.700305] LustreError: 155902:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3538.703480] LustreError: 155902:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3538.768510] Lustre: server umount lustre-MDT0001 complete [ 3542.340302] Lustre: DEBUG MARKER: == conf-sanity test 47a: server restart does not lose lru_resize ========================================================== 20:35:59 (1763084159) [ 3548.716228] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 3552.938955] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 3557.228716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3559.918630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3562.708143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3565.846954] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3568.761271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3568.794440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3568.903676] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3568.919246] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3568.961430] Lustre: lustre-MDT0000: new disk, initializing [ 3568.996194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3569.004926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3570.727342] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3576.161315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3576.198869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3576.250336] Lustre: 158890:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3576.286861] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3576.289991] Lustre: Skipped 1 previous similar message [ 3576.336461] Lustre: lustre-MDT0001: new disk, initializing [ 3576.387270] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3576.410829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3576.418500] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3578.452527] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3582.517498] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3584.081333] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3587.273926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3587.307959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3587.422158] Lustre: lustre-OST0000: new disk, initializing [ 3587.424734] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3587.460770] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3589.848084] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3593.495468] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3593.733783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3593.738613] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3593.776718] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3601.152140] Lustre: Failing over lustre-OST0000 [ 3601.187133] LustreError: 161332:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3601.190285] LustreError: 161332:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3601.238467] Lustre: server umount lustre-OST0000 complete [ 3615.059113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3615.200417] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3615.206186] Lustre: Skipped 1 previous similar message [ 3615.212756] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3615.222215] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3615.406973] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3617.721710] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3619.540374] Lustre: Failing over lustre-MDT0000 [ 3619.620096] LustreError: 162304:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3619.623534] LustreError: 162304:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3619.661921] Lustre: server umount lustre-MDT0000 complete [ 3620.320303] Lustre: 150774:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763084220/real 1763084220] req@0000000015c7a41b x1848727663865088/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1763084236 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3620.321252] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3620.339051] Lustre: Skipped 2 previous similar messages [ 3621.343488] Lustre: 150773:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763084220/real 1763084220] req@000000000e43ab2b x1848727663864960/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1763084236 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3625.440470] LustreError: 158900:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3625.448057] LustreError: 158900:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 3625.503211] Lustre: 150775:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763084225/real 1763084225] req@00000000c39a439f x1848727663866240/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1763084241 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3630.559196] Lustre: 150776:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763084230/real 1763084230] req@00000000c53daa64 x1848727663867392/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1763084246 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3630.560161] LustreError: 158900:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3630.572516] LustreError: 158900:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3633.491184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3633.558965] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3633.563069] Lustre: 158892:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.203.103@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3633.649396] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 3633.699430] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3635.311501] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3635.875670] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3638.756610] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3638.764375] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 3638.767283] Lustre: Skipped 1 previous similar message [ 3638.785538] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3643.873120] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3643.874095] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3643.877627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3643.878136] LustreError: Skipped 1 previous similar message [ 3643.888048] Lustre: Skipped 1 previous similar message [ 3646.011235] LustreError: 163186:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3646.018195] LustreError: 163186:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3646.076643] Lustre: server umount lustre-OST0000 complete [ 3648.996132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3649.000175] Lustre: Skipped 1 previous similar message [ 3649.001650] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3659.232437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3659.235188] Lustre: Skipped 3 previous similar messages [ 3662.303164] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3662.356233] LustreError: 163389:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3662.359059] LustreError: 163389:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3662.396369] Lustre: server umount lustre-MDT0000 complete [ 3663.958231] LustreError: 159670:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763084280 with bad export cookie 4020181992264405494 [ 3663.961623] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3663.963292] LustreError: 159670:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3664.113445] Lustre: server umount lustre-MDT0001 complete [ 3668.556961] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 3670.039493] Key type lgssc unregistered [ 3670.191785] LNet: 164122:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3670.196418] LNetError: 164122:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3670.207354] LNet: Removed LNI 192.168.203.103@tcp [ 3670.555190] Key type .llcrypt unregistered [ 3670.556984] Key type ._llcrypt unregistered [ 3692.951675] Key type ._llcrypt registered [ 3692.953193] Key type .llcrypt registered [ 3693.032485] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 3693.417265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3693.438694] alg: No test for adler32 (adler32-zlib) [ 3694.380083] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 3694.505762] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 3696.111157] Key type lgssc registered [ 3696.620980] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3700.553390] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3700.569655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3701.810053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3703.883966] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3707.947865] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3710.469557] LustreError: 165650:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3710.474549] LustreError: 165650:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3710.478332] LustreError: 165650:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -5 [ 3710.563154] LustreError: 166172:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3710.649969] Lustre: server umount lustre-MDT0000 complete [ 3717.063502] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 3720.940055] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 3725.108251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3727.764263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3730.166094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3732.668397] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3735.216958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3735.241853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3735.330433] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3735.345519] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3735.388963] Lustre: lustre-MDT0000: new disk, initializing [ 3735.422221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3735.432065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3736.893363] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3741.435957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3741.457411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3741.496799] Lustre: 168811:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3741.523337] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3741.525510] Lustre: Skipped 1 previous similar message [ 3741.561853] Lustre: lustre-MDT0001: new disk, initializing [ 3741.599293] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3741.615485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3741.620340] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3743.096778] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3746.617770] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3748.208917] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3750.975775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3751.016314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3751.132377] Lustre: lustre-OST0000: new disk, initializing [ 3751.135731] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3751.167436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3751.379830] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3751.386087] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3751.440429] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3753.305688] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3756.690042] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3758.642412] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 3759.304068] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 20:39:36 (1763084376) [ 3838.432933] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3838.438808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3838.441834] Lustre: Skipped 2 previous similar messages [ 3843.551602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3843.554305] Lustre: Skipped 2 previous similar messages [ 3844.604709] LustreError: 172313:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3844.607759] LustreError: 172313:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3844.658355] Lustre: server umount lustre-MDT0000 complete [ 3846.046496] LustreError: 168802:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763084462 with bad export cookie 2305394792700434924 [ 3846.047658] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3846.051553] LustreError: 168802:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3848.671690] LustreError: 168817:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3848.671892] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3848.679585] LustreError: 168817:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3848.684730] Lustre: Skipped 2 previous similar messages [ 3848.686033] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3851.679771] LustreError: 168818:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3851.679792] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3852.156177] LustreError: 172515:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3852.158017] LustreError: 172515:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3852.202404] Lustre: server umount lustre-MDT0001 complete [ 3859.694774] LustreError: 172716:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3859.696744] LustreError: 172716:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3859.747975] Lustre: server umount lustre-OST0000 complete [ 3863.271783] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 3867.073933] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 3871.015772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3873.611878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3875.851884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3878.164745] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3880.684167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3880.708686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3880.792149] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3880.803653] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3880.842853] Lustre: lustre-MDT0000: new disk, initializing [ 3880.871986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3880.878517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3882.359849] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3887.145257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3887.173918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3887.216049] Lustre: 175050:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3887.246388] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3887.248904] Lustre: Skipped 1 previous similar message [ 3887.288749] Lustre: lustre-MDT0001: new disk, initializing [ 3887.337612] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3887.341599] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3888.830313] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3892.162914] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3893.544101] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3896.078995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3896.110742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3896.224669] Lustre: lustre-OST0000: new disk, initializing [ 3896.227954] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3896.263692] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3896.266497] Lustre: Skipped 1 previous similar message [ 3896.388302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3896.392567] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3896.430154] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3898.343619] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3901.569733] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3903.135749] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3903.233668] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3904.730328] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3904.818568] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3906.529795] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3906.530476] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3906.532824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3906.534968] Lustre: Skipped 1 previous similar message [ 3906.538784] Lustre: Skipped 1 previous similar message [ 3911.599321] LustreError: 177989:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3911.602372] LustreError: 177989:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3911.661470] Lustre: server umount lustre-OST0000 complete [ 3916.769119] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3916.773332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3916.774907] Lustre: Skipped 1 previous similar message [ 3919.478553] Lustre: server umount lustre-MDT0000 complete [ 3921.192673] LustreError: 175042:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763084537 with bad export cookie 2305394792700593740 [ 3921.195734] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3921.198034] LustreError: 175042:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3921.279728] LustreError: 178393:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3921.282872] LustreError: 178393:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 3921.367439] Lustre: server umount lustre-MDT0001 complete [ 3924.992036] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 20:42:22 (1763084542) [ 3931.774280] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 3935.723133] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 3939.917269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3942.692119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3945.445493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3948.134163] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3951.055946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3951.096486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3951.208966] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3951.223416] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3951.262948] Lustre: lustre-MDT0000: new disk, initializing [ 3951.294889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3951.302595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3952.785552] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3957.174984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3957.205562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3957.244850] Lustre: 181379:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3957.352613] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3958.644299] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3961.947961] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3963.287486] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3965.800946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3965.825877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3967.968849] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 3971.203331] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3975.162745] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3975.201485] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3980.257699] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3980.259722] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3980.262332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3980.266470] Lustre: Skipped 1 previous similar message [ 3980.268913] Lustre: Skipped 1 previous similar message [ 3984.047186] LustreError: 183926:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3984.050561] LustreError: 183926:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3984.095883] Lustre: server umount lustre-OST0000 complete [ 3987.936789] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3987.939943] LustreError: Skipped 1 previous similar message [ 3990.496814] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3990.502060] Lustre: Skipped 1 previous similar message [ 3991.743706] Lustre: server umount lustre-MDT0000 complete [ 3993.145191] LustreError: 181371:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763084609 with bad export cookie 2305394792700594608 [ 3993.148422] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3993.148586] LustreError: 181371:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3993.266944] Lustre: server umount lustre-MDT0001 complete [ 3996.290473] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 20:43:33 (1763084613) [ 4002.137969] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 4005.790732] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 4009.589594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4012.087278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4014.381662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4016.949150] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4019.715285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4019.744315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4019.850597] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4019.865505] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4019.868414] Lustre: Skipped 3 previous similar messages [ 4019.911168] Lustre: lustre-MDT0000: new disk, initializing [ 4019.913044] Lustre: Skipped 2 previous similar messages [ 4019.951681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4019.954892] Lustre: Skipped 2 previous similar messages [ 4019.963082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4019.966058] Lustre: Skipped 2 previous similar messages [ 4021.580629] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4026.462297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4026.494569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4026.539813] Lustre: 187314:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4026.661656] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4028.141774] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4031.718026] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4033.143794] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4035.640437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4035.680760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4037.923148] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4041.205455] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4041.771608] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4051.940142] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4051.942267] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4051.944972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4051.944979] Lustre: Skipped 4 previous similar messages [ 4051.949825] Lustre: Skipped 1 previous similar message [ 4054.703291] LustreError: 189862:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4054.707197] LustreError: 189862:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4054.758744] Lustre: server umount lustre-OST0000 complete [ 4062.431164] Lustre: server umount lustre-MDT0000 complete [ 4063.762564] LustreError: 188111:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763084680 with bad export cookie 2305394792700595679 [ 4063.767296] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4063.767916] LustreError: 188111:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4063.895850] Lustre: server umount lustre-MDT0001 complete [ 4067.720842] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 4068.866951] Key type lgssc unregistered [ 4069.001448] LNet: 190796:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4069.003970] LNetError: 190796:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4069.018323] LNet: Removed LNI 192.168.203.103@tcp [ 4069.345099] Key type .llcrypt unregistered [ 4069.346723] Key type ._llcrypt unregistered [ 4079.217633] Key type ._llcrypt registered [ 4079.219176] Key type .llcrypt registered [ 4079.261274] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 4079.615837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4079.671739] alg: No test for adler32 (adler32-zlib) [ 4080.542079] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 4080.636820] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 4082.223233] Key type lgssc registered [ 4082.648198] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4085.457628] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4085.464918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4086.617469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4087.907871] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4090.904698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4091.043232] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4092.483315] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4094.354695] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4095.728152] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4098.154408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4098.280809] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4100.274498] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4102.157906] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4111.590376] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4127.200932] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4127.201599] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4127.203718] LustreError: Skipped 1 previous similar message [ 4127.208282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4127.211028] Lustre: Skipped 1 previous similar message [ 4130.413140] LustreError: 194437:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4130.450932] Lustre: server umount lustre-OST0000 complete [ 4132.322275] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4132.322470] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4132.323218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4132.327186] Lustre: Skipped 2 previous similar messages [ 4137.440753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4137.442977] Lustre: Skipped 2 previous similar messages [ 4142.560682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4142.563179] Lustre: Skipped 1 previous similar message [ 4146.143114] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4146.182344] LustreError: 194641:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4146.185491] LustreError: 194641:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4146.228374] Lustre: server umount lustre-MDT0000 complete [ 4147.680553] LustreError: 192171:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4147.687145] LustreError: 192171:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4147.719631] LustreError: 194643:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763084763 with bad export cookie 6784386314685047509 [ 4147.723729] LustreError: 194643:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4147.724231] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4147.791717] LustreError: 194845:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4147.794216] LustreError: 194845:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4147.856637] Lustre: server umount lustre-MDT0001 complete [ 4152.106515] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 4153.330299] Key type lgssc unregistered [ 4153.479743] LNet: 195376:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4153.484206] LNetError: 195376:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4153.493562] LNet: Removed LNI 192.168.203.103@tcp [ 4153.834074] Key type .llcrypt unregistered [ 4153.835609] Key type ._llcrypt unregistered [ 4163.832113] Key type ._llcrypt registered [ 4163.833540] Key type .llcrypt registered [ 4163.880986] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 4164.211328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4164.228044] alg: No test for adler32 (adler32-zlib) [ 4165.093500] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 4165.187507] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 4166.775126] Key type lgssc registered [ 4167.195917] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4170.198459] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4170.207036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4171.419441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4172.877276] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4176.170941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4176.308977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4177.752135] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4179.650120] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4180.911555] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4183.484458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4183.614589] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4185.827695] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4187.882039] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4195.942899] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4201.442168] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4201.442331] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4201.443884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4201.454148] Lustre: Skipped 1 previous similar message [ 4203.694746] Lustre: lustre-OST0000: Not available for connect from 192.168.203.3@tcp (stopping) [ 4203.698221] Lustre: Skipped 1 previous similar message [ 4206.562025] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4206.564719] Lustre: Skipped 1 previous similar message [ 4207.214302] LustreError: 199018:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4207.259083] Lustre: server umount lustre-OST0000 complete [ 4209.437943] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8903d9351800.ost_server_uuid 50 [ 4210.165834] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 4221.920378] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4221.921160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4221.925683] Lustre: Skipped 1 previous similar message [ 4226.494703] LustreError: 199366:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4226.497452] LustreError: 199366:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4226.523717] Lustre: server umount lustre-MDT0000 complete [ 4227.040580] LustreError: 198909:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4227.051073] LustreError: 198909:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4228.012635] LustreError: 196732:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763084844 with bad export cookie 3681936853546239942 [ 4228.015477] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4228.017370] LustreError: 196732:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4228.102314] LustreError: 199568:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4228.105785] LustreError: 199568:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4228.176894] Lustre: server umount lustre-MDT0001 complete [ 4231.622155] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 20:47:28 (1763084848) [ 4234.098308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4234.266753] LustreError: 200239:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4234.289739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4235.733208] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4238.890940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4239.031480] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4240.387940] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4242.398073] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4243.718949] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4246.250660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4248.458125] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4250.424501] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4253.119070] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4253.150113] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4253.224054] Lustre: lustre-OST0001: new disk, initializing [ 4253.226809] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4253.261100] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4253.263859] Lustre: Skipped 1 previous similar message [ 4253.411198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4253.415164] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4253.445321] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4255.380542] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4258.746452] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4259.561302] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4259.809116] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4259.810878] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4259.815469] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4259.818676] Lustre: Skipped 1 previous similar message [ 4264.929677] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4264.931358] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4265.774081] LustreError: 203444:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4265.777272] LustreError: 203444:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4265.797189] Lustre: server umount lustre-OST0000 complete [ 4267.987924] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4268.082053] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4269.220297] LustreError: 201833:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4269.765687] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4269.861749] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4274.339893] LustreError: 201836:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4274.346186] LustreError: 201836:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4280.290123] LustreError: 201833:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4280.290808] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4280.292348] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4280.292354] Lustre: Skipped 3 previous similar messages [ 4280.296706] LustreError: 201833:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4280.301815] Lustre: Skipped 1 previous similar message [ 4285.995453] LustreError: 204146:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 4285.997449] LustreError: 204146:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4286.040491] Lustre: server umount lustre-OST0001 complete [ 4290.528811] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4290.528962] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4293.534942] Lustre: server umount lustre-MDT0000 complete [ 4294.863503] LustreError: 202747:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763084911 with bad export cookie 3681936853546240999 [ 4294.866637] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4294.868896] LustreError: 202747:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4294.936952] LustreError: 204550:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 4294.939250] LustreError: 204550:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 4294.996225] Lustre: server umount lustre-MDT0001 complete [ 4296.992747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4298.478584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4299.929389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4301.377069] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4303.828526] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 20:48:40 (1763084920) [ 4306.210659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4306.310277] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4306.322055] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4306.397294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4307.801648] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4310.824924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4310.858542] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4310.873520] Lustre: 206039:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4312.347877] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4314.256130] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4315.399894] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4317.609142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4317.673398] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4319.547430] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4321.205507] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4321.736944] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 4323.469350] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4323.492217] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4325.428157] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4327.196883] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4328.930027] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4328.931041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4328.933430] Lustre: Skipped 2 previous similar messages [ 4328.937494] Lustre: Skipped 6 previous similar messages [ 4330.511988] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 4334.190106] LustreError: 209223:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4334.192346] LustreError: 209223:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 4334.212380] Lustre: server umount lustre-OST0000 complete [ 4335.588313] LustreError: 208285:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4335.592684] LustreError: 208285:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4345.825689] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4351.059568] Lustre: server umount lustre-OST0001 complete [ 4358.553287] Lustre: server umount lustre-MDT0000 complete [ 4359.873088] LustreError: 206031:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763084976 with bad export cookie 3681936853546242385 [ 4359.876632] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4359.878487] LustreError: 206031:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4359.997536] Lustre: server umount lustre-MDT0001 complete [ 4361.950996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4363.336571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4364.671363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4365.994526] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4368.290596] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 20:49:45 (1763084985) [ 4374.090221] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 4377.656584] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 4381.214268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4383.412616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4385.582023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4387.744975] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4390.052371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4390.074910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4390.167838] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4390.183262] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4390.222819] Lustre: lustre-MDT0000: new disk, initializing [ 4390.252361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4390.255330] Lustre: Skipped 3 previous similar messages [ 4390.262228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4391.683022] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4395.991168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4396.014419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4396.050780] Lustre: 213654:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4396.077229] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4396.079935] Lustre: Skipped 1 previous similar message [ 4396.120545] Lustre: lustre-MDT0001: new disk, initializing [ 4396.167275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4396.171611] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4397.591157] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4400.901972] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4402.188796] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4404.614472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4404.635870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4404.741884] Lustre: lustre-OST0000: new disk, initializing [ 4404.744455] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4406.942102] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4407.612829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4407.616673] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4407.652755] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4410.117416] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4411.506629] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4411.582763] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4412.933697] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4413.021896] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4415.729098] LustreError: 216592:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4415.732092] LustreError: 216592:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 4415.783358] Lustre: server umount lustre-OST0000 complete [ 4418.016688] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4418.017280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4418.020582] Lustre: Skipped 5 previous similar messages [ 4418.022050] Lustre: Skipped 9 previous similar messages [ 4427.266465] Lustre: server umount lustre-MDT0000 complete [ 4428.256319] LustreError: 213665:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4428.262277] LustreError: 213665:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 4428.600572] LustreError: 213646:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763085044 with bad export cookie 3681936853546244128 [ 4428.603730] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4428.604545] LustreError: 213646:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4438.036364] Lustre: server umount lustre-MDT0001 complete [ 4440.898976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4442.427717] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4445.314733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4446.780205] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4448.560198] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4449.665552] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4451.925243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4453.958117] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4455.856150] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4457.336817] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4467.645635] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec [ 4468.969413] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4469.040650] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4470.751793] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4470.753923] LustreError: Skipped 1 previous similar message [ 4475.726793] Lustre: server umount lustre-OST0000 complete [ 4477.954704] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4479.547972] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4503.229985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4503.334078] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4503.337836] Lustre: Skipped 5 previous similar messages [ 4505.221692] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4507.009517] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4507.629595] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4513.762497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4513.764503] Lustre: Skipped 6 previous similar messages [ 4514.730088] LustreError: 221921:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4514.732620] LustreError: 221921:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 4514.764557] Lustre: server umount lustre-OST0000 complete [ 4517.343848] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4517.345985] LustreError: Skipped 1 previous similar message [ 4517.347122] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4517.350133] Lustre: Skipped 2 previous similar messages [ 4522.131095] Lustre: server umount lustre-MDT0000 complete [ 4522.976174] LustreError: 218184:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4522.980010] LustreError: 218184:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4523.314862] LustreError: 217453:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763085139 with bad export cookie 3681936853546244996 [ 4523.317819] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4523.319480] LustreError: 217453:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4523.427555] Lustre: server umount lustre-MDT0001 complete [ 4526.118909] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 20:52:23 (1763085143) [ 4528.267347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4529.737477] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4532.541761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4533.978816] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4535.787904] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4536.999722] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4539.322381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4541.328693] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4543.172879] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4544.582476] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4553.886207] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 4555.129771] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4555.204352] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4557.381844] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4557.408823] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4557.457574] Lustre: lustre-OST0001: new disk, initializing [ 4557.459652] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4559.413213] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4562.498896] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4563.926395] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4567.548173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4567.550765] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4567.579810] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4568.087734] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 4569.419933] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4569.497258] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4572.641291] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4572.643815] LustreError: Skipped 1 previous similar message [ 4576.256990] Lustre: server umount lustre-OST0001 complete [ 4578.222382] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4578.308709] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4579.553358] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4579.621195] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4588.004059] LustreError: 227523:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4588.009856] LustreError: 227523:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 4603.296425] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4605.450306] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4607.267307] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4607.847856] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4608.487339] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4608.491229] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4608.491307] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4608.498525] Lustre: Skipped 1 previous similar message [ 4614.586319] Lustre: server umount lustre-OST0001 complete [ 4632.728492] Lustre: server umount lustre-OST0000 complete [ 4640.225080] Lustre: server umount lustre-MDT0000 complete [ 4641.594754] LustreError: 222980:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763085257 with bad export cookie 3681936853546245927 [ 4641.597677] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4641.599105] LustreError: 222980:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4647.741948] LustreError: 229572:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 4647.745020] LustreError: 229572:0:(obd_class.h:479:obd_check_dev()) Skipped 37 previous similar messages [ 4647.795981] Lustre: server umount lustre-MDT0001 complete [ 4649.616094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4650.895096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4652.150487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4653.431459] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4655.663502] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 20:54:32 (1763085272) [ 4657.867853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4657.971870] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4657.984305] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4658.062402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4658.065092] Lustre: Skipped 5 previous similar messages [ 4659.398037] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4662.308321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4662.336031] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4662.343826] Lustre: 231062:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4663.740823] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4665.515841] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4666.667249] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4668.873017] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4668.939372] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4670.920940] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4672.791063] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4682.677759] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4682.699512] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4684.719235] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4686.516566] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4689.134828] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 4690.439111] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4690.520716] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4691.824366] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4691.919336] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4693.111350] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8903c590f800.ost_server_uuid 50 [ 4693.637493] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8903c590f800.ost_server_uuid in FULL state after 0 sec [ 4693.997767] Lustre: Permanently deactivating lustre-OST0001 [ 4694.003628] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 4694.782959] Lustre: Permanently reactivating lustre-OST0001 [ 4700.128444] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4700.131522] LustreError: Skipped 2 previous similar messages [ 4700.133143] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4700.134478] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4700.137748] Lustre: Skipped 10 previous similar messages [ 4700.140074] Lustre: Skipped 14 previous similar messages [ 4701.753673] Lustre: server umount lustre-OST0001 complete [ 4709.062547] Lustre: server umount lustre-OST0000 complete [ 4716.378433] Lustre: server umount lustre-MDT0000 complete [ 4717.620313] LustreError: 231054:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763085333 with bad export cookie 3681936853546247201 [ 4717.623864] LustreError: 231054:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4717.624253] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4717.735592] Lustre: server umount lustre-MDT0001 complete [ 4719.658881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4720.820793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4721.988996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4723.213690] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4725.307527] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 20:55:42 (1763085342) [ 4725.621624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4727.753542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4727.849060] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4727.858650] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4727.861097] Lustre: Skipped 5 previous similar messages [ 4729.259145] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4732.143744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4732.173115] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4732.181974] Lustre: 237201:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4733.585061] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4735.334229] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4736.512298] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4738.730904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4738.757481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4738.835152] Lustre: Permanently deactivating lustre-OST0000 [ 4738.840123] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 4740.729478] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4742.257962] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4742.261608] LustreError: 239352:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0001: cannot cleanup orphans: rc = -108 [ 4742.513146] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4745.022577] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4747.166800] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4749.315233] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4752.406294] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 4755.018491] Lustre: Permanently reactivating lustre-OST0000 [ 4755.023914] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 4755.026733] Lustre: Skipped 2 previous similar messages [ 4760.047601] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 4760.049906] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4760.053945] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4760.070356] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 4760.077579] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4764.030630] Lustre: server umount lustre-OST0001 complete [ 4767.713881] LustreError: 238829:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4767.719077] LustreError: 238829:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 17 previous similar messages [ 4771.290469] Lustre: server umount lustre-OST0000 complete [ 4772.804560] Lustre: server umount lustre-MDT0000 complete [ 4774.136287] LustreError: 240352:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763085390 with bad export cookie 3681936853546248825 [ 4774.141555] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4774.278944] Lustre: server umount lustre-MDT0001 complete [ 4777.898424] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 4778.932851] Key type lgssc unregistered [ 4779.074309] LNet: 241595:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4779.076298] LNetError: 241595:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4779.082284] LNet: Removed LNI 192.168.203.103@tcp [ 4779.378046] Key type .llcrypt unregistered [ 4779.379089] Key type ._llcrypt unregistered [ 4788.122114] Key type ._llcrypt registered [ 4788.123117] Key type .llcrypt registered [ 4788.169137] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 4788.516730] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4788.544094] alg: No test for adler32 (adler32-zlib) [ 4789.400776] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 4789.480590] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 4791.063155] Key type lgssc registered [ 4791.509646] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4792.756579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4795.017763] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4795.023601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4796.199634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4797.531559] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4800.530366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4800.557235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4800.591736] Lustre: Found index 1 for lustre-MDT0001, updating log [ 4800.600862] Lustre: 243011:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4800.604664] Lustre: Permanently deactivating lustre-MDT0001 [ 4800.610327] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 4800.728859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4801.999610] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4803.811132] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4805.093576] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4805.615482] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 4805.618896] LustreError: 243053:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4805.622981] LustreError: 243053:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4805.626879] LustreError: 243053:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 1, failed: rc = -108 [ 4807.480727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4807.606482] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4809.616530] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4810.661227] LustreError: 244642:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4810.668809] LustreError: 244642:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4811.532766] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4813.933344] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4814.017724] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4816.122782] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4817.967711] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4819.430720] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 4821.493558] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 4821.496914] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 4823.650622] Lustre: Permanently reactivating lustre-MDT0001 [ 4823.656838] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 4823.659123] Lustre: Skipped 2 previous similar messages [ 4828.332471] Lustre: lustre-MDT0001: Client 46dc2217-f05d-4221-a935-24c0c6e9e151 (at 192.168.203.3@tcp) reconnecting [ 4829.972914] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4830.552565] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4832.186849] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 4832.260666] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 4833.550759] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 4833.629103] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 4834.446278] Lustre: Permanently deactivating lustre-MDT0001 [ 4834.451989] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 4834.455085] Lustre: Skipped 2 previous similar messages [ 4839.935927] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 4842.208331] LustreError: 247089:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 4842.239823] Lustre: server umount lustre-MDT0000 complete [ 4843.450360] LustreError: 243002:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763085459 with bad export cookie 10844521587881825270 [ 4843.453914] LustreError: 243002:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4843.454340] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4847.072304] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4847.073172] LustreError: 244379:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4847.076916] Lustre: Skipped 1 previous similar message [ 4848.096229] LustreError: 246142:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4848.100720] LustreError: 246142:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4849.601100] LustreError: 247289:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 4849.603548] LustreError: 247289:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4849.649981] Lustre: server umount lustre-MDT0001 complete [ 4851.010071] LustreError: 247490:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 4851.011996] LustreError: 247490:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4851.028551] Lustre: server umount lustre-OST0000 complete [ 4854.893466] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 20:57:52 (1763085472) [ 4858.533381] LustreError: 248494:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4858.536034] LustreError: 248494:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 4858.575620] Lustre: server umount lustre-OST0001 complete [ 4860.680186] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 4864.072505] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 4867.461178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4869.581129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4871.630891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4873.913727] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4876.346371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4876.366123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4876.441941] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4876.446834] Lustre: Skipped 2 previous similar messages [ 4876.456234] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4876.487678] Lustre: lustre-MDT0000: new disk, initializing [ 4876.507476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4876.513417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4877.737387] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4881.669197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4881.689751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4881.720356] Lustre: 250675:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4881.741197] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4881.742674] Lustre: Skipped 1 previous similar message [ 4881.775471] Lustre: lustre-MDT0001: new disk, initializing [ 4881.812277] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4881.815655] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4883.012326] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4886.033318] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4887.280121] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4889.567109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4889.593756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4889.692292] Lustre: lustre-OST0000: new disk, initializing [ 4889.695340] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4889.726828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4889.730356] Lustre: Skipped 1 previous similar message [ 4891.636651] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4894.864432] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4896.249509] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4896.253621] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4896.282980] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4901.064342] LustreError: 250680:0:(mdt_handler.c:3230:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4905.258130] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4905.283388] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4905.333747] Lustre: lustre-OST0001: new disk, initializing [ 4905.336204] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4907.144637] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 4910.102165] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4910.602902] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4911.159126] LustreError: 250680:0:(mdt_handler.c:3230:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4911.167065] LustreError: 250680:0:(mdt_handler.c:3230:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4915.199564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4915.203381] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4915.231294] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4921.255089] LustreError: 250680:0:(mdt_handler.c:3230:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4924.072309] Lustre: lustre-OST0001: Not available for connect from 192.168.203.3@tcp (stopping) [ 4925.414293] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4925.415063] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4925.417177] LustreError: Skipped 1 previous similar message [ 4925.423705] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4925.426444] Lustre: Skipped 1 previous similar message [ 4928.045627] LustreError: 254191:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 4928.049984] LustreError: 254191:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4928.072811] Lustre: server umount lustre-OST0001 complete [ 4930.532261] LustreError: 252405:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4930.539239] LustreError: 252405:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4935.653311] LustreError: 253028:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4935.658411] LustreError: 253028:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4940.769688] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4940.772030] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4940.773264] Lustre: Skipped 2 previous similar messages [ 4945.892659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4945.893387] LustreError: 252404:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4945.895494] Lustre: Skipped 2 previous similar messages [ 4945.901386] LustreError: 252404:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4946.284099] LustreError: 254413:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4946.286601] LustreError: 254413:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4946.328979] Lustre: server umount lustre-OST0000 complete [ 4948.447949] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4948.451426] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4951.009732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4951.011507] Lustre: Skipped 2 previous similar messages [ 4953.893563] Lustre: server umount lustre-MDT0000 complete [ 4955.207212] LustreError: 250665:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763085571 with bad export cookie 10844521587881827867 [ 4955.211473] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4955.211894] LustreError: 250665:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4961.392931] Lustre: server umount lustre-MDT0001 complete [ 4964.991134] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 4966.040859] Key type lgssc unregistered [ 4966.174455] LNet: 255347:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4966.177648] LNetError: 255347:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4966.185361] LNet: Removed LNI 192.168.203.103@tcp [ 4966.489103] Key type .llcrypt unregistered [ 4966.490954] Key type ._llcrypt unregistered [ 4977.473885] Key type ._llcrypt registered [ 4977.474972] Key type .llcrypt registered [ 4977.511644] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 4983.676527] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 4984.136714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4984.153621] alg: No test for adler32 (adler32-zlib) [ 4985.017948] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 4985.100789] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 4986.679166] Key type lgssc registered [ 4987.038530] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4989.970987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4992.244941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4994.364707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4996.624051] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4998.707269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4998.726623] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4998.731100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4999.804967] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4999.814552] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4999.847658] Lustre: lustre-MDT0000: new disk, initializing [ 4999.868715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4999.873617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5001.022621] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5004.822203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5004.844414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5004.874960] Lustre: 259268:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5004.897996] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5004.899692] Lustre: Skipped 1 previous similar message [ 5004.936772] Lustre: lustre-MDT0001: new disk, initializing [ 5004.969278] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5004.982074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5004.985294] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5006.110756] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5009.017421] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5010.049407] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5012.102056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5012.124028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5012.208863] Lustre: lustre-OST0000: new disk, initializing [ 5012.210414] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5012.233938] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5014.168440] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5017.079557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5017.082347] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5017.112043] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5017.197880] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5018.419860] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5018.487130] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5019.713928] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5019.786750] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5022.177386] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5022.178203] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5022.178923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5022.180197] LustreError: Skipped 1 previous similar message [ 5022.182962] Lustre: Skipped 1 previous similar message [ 5026.478104] LustreError: 262205:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5026.519051] Lustre: server umount lustre-OST0000 complete [ 5030.879945] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5030.882229] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5030.885919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5030.887357] Lustre: Skipped 1 previous similar message [ 5032.416516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5032.416848] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5037.536135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5037.537729] Lustre: Skipped 2 previous similar messages [ 5042.143117] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5042.178345] LustreError: 262407:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5042.181626] LustreError: 262407:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5042.213151] Lustre: server umount lustre-MDT0000 complete [ 5043.423310] LustreError: 259260:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763085659 with bad export cookie 16368981752211081839 [ 5043.426665] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5043.426926] LustreError: 259260:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5043.489536] LustreError: 262610:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5043.491684] LustreError: 262610:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5043.540750] Lustre: server umount lustre-MDT0001 complete [ 5046.271836] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 21:01:03 (1763085663) [ 5048.509130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5048.672081] LustreError: 263292:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5048.678974] LustreError: 263292:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5048.699471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5050.040500] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5052.900849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5053.016130] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5054.298128] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5056.128521] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5057.453737] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5059.897251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5062.103680] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5063.939844] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5114.336352] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5114.339762] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5114.346268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5117.740076] LustreError: 265632:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5117.741966] LustreError: 265632:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5117.791821] Lustre: server umount lustre-OST0000 complete [ 5119.265792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 5123.377509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5123.502496] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5123.505679] Lustre: Skipped 1 previous similar message [ 5125.371724] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5127.118741] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5134.756327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5134.764305] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5139.876469] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5139.880410] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5160.419397] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5160.420188] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5160.424778] Lustre: Skipped 1 previous similar message [ 5160.426693] LustreError: Skipped 1 previous similar message [ 5160.431248] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5160.433801] Lustre: Skipped 3 previous similar messages [ 5166.317912] LustreError: 267296:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5166.320197] LustreError: 267296:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5166.365566] Lustre: server umount lustre-OST0000 complete [ 5170.657071] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5170.657408] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5170.659528] LustreError: Skipped 1 previous similar message [ 5170.663617] Lustre: Skipped 2 previous similar messages [ 5173.776687] Lustre: server umount lustre-MDT0000 complete [ 5175.005308] LustreError: 263273:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763085791 with bad export cookie 16368981752211082707 [ 5175.008805] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5175.009522] LustreError: 263273:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5175.081381] LustreError: 267702:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5175.084133] LustreError: 267702:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 5175.133789] Lustre: server umount lustre-MDT0001 complete [ 5178.594648] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 5179.579039] Key type lgssc unregistered [ 5179.715309] LNet: 268232:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5179.717519] LNetError: 268232:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5179.731261] LNet: Removed LNI 192.168.203.103@tcp [ 5180.033080] Key type .llcrypt unregistered [ 5180.034138] Key type ._llcrypt unregistered [ 5189.296955] Key type ._llcrypt registered [ 5189.298566] Key type .llcrypt registered [ 5189.340749] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 5189.829067] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5189.840560] alg: No test for adler32 (adler32-zlib) [ 5190.705599] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 5190.794044] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 5192.375124] Key type lgssc registered [ 5192.764348] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5195.444373] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5195.449883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5196.616640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5198.013060] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5201.015720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5201.152418] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5202.501322] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5204.370220] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5205.675936] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5208.036768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5208.153149] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5210.071452] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5211.923247] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5221.478860] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 5232.097809] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5232.098530] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5232.100168] LustreError: Skipped 1 previous similar message [ 5232.100804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5232.103527] Lustre: Skipped 1 previous similar message [ 5232.107581] Lustre: Skipped 1 previous similar message [ 5236.078110] LustreError: 272662:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5236.111915] Lustre: server umount lustre-OST0000 complete [ 5242.336912] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5242.337046] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5242.337761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5242.347148] Lustre: Skipped 1 previous similar message [ 5243.518583] LustreError: 272864:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5243.520819] LustreError: 272864:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5243.549107] Lustre: server umount lustre-MDT0000 complete [ 5244.746234] LustreError: 271240:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763085861 with bad export cookie 14202247963468682543 [ 5244.748953] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5244.751907] LustreError: 271240:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5244.825143] LustreError: 273066:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5244.828245] LustreError: 273066:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5244.882317] Lustre: server umount lustre-MDT0001 complete [ 5248.287206] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 5249.355804] Key type lgssc unregistered [ 5249.490706] LNet: 273597:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5249.494674] LNetError: 273597:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5249.501322] LNet: Removed LNI 192.168.203.103@tcp [ 5249.834105] Key type .llcrypt unregistered [ 5249.835560] Key type ._llcrypt unregistered [ 5257.069482] Key type ._llcrypt registered [ 5257.070834] Key type .llcrypt registered [ 5257.108391] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 5257.510633] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5257.567129] alg: No test for adler32 (adler32-zlib) [ 5258.434770] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 5258.524910] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 5260.111171] Key type lgssc registered [ 5260.542093] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5263.335713] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5263.344304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5264.513723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5265.965068] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5268.895808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5269.021923] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5270.432031] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5272.453794] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5273.769176] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5276.117447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5276.247913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5278.180200] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5279.968098] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5290.599555] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5295.071815] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5295.073727] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5295.077192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5296.098893] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5296.099673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5296.104826] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5296.108608] Lustre: Skipped 1 previous similar message [ 5296.365160] LustreError: 277187:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5296.402651] Lustre: server umount lustre-OST0000 complete [ 5299.680241] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5299.683810] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5299.690085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5299.843293] LustreError: 277390:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5299.845396] LustreError: 277390:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5299.874098] Lustre: server umount lustre-MDT0000 complete [ 5301.125587] LustreError: 277391:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763085917 with bad export cookie 8396224227444273793 [ 5301.129728] LustreError: 277391:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5301.130109] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5301.197732] LustreError: 277591:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5301.199521] LustreError: 277591:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5301.256243] Lustre: server umount lustre-MDT0001 complete [ 5304.775860] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 5305.777346] Key type lgssc unregistered [ 5305.911340] LNet: 278122:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5305.913634] LNetError: 278122:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5305.921283] LNet: Removed LNI 192.168.203.103@tcp [ 5306.207280] Key type .llcrypt unregistered [ 5306.209048] Key type ._llcrypt unregistered [ 5315.655511] Key type ._llcrypt registered [ 5315.656668] Key type .llcrypt registered [ 5315.698115] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 5316.107319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5316.164741] alg: No test for adler32 (adler32-zlib) [ 5317.016846] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 5317.113245] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 5318.703174] Key type lgssc registered [ 5319.050575] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5321.792406] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5321.799217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5322.952894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5324.206536] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5327.133041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5327.253253] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5328.526106] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5330.308252] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5331.554398] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5333.885996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5334.002278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5335.869875] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5337.735608] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5348.326628] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5361.121122] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5361.123163] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5361.126485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5362.655794] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5362.657654] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5362.660916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5363.680877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5363.682678] Lustre: Skipped 1 previous similar message [ 5366.316491] LustreError: 282537:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5366.350264] Lustre: server umount lustre-OST0000 complete [ 5368.288216] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5368.290283] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5368.295102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5373.629780] LustreError: 282739:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5373.633725] LustreError: 282739:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5373.659309] Lustre: server umount lustre-MDT0000 complete [ 5373.920460] LustreError: 279496:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5373.924962] LustreError: 279496:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5374.905193] LustreError: 280216:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763085991 with bad export cookie 18076133949979564650 [ 5374.909139] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5374.909611] LustreError: 280216:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5374.973904] LustreError: 282942:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5374.975984] LustreError: 282942:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5375.023311] Lustre: server umount lustre-MDT0001 complete [ 5378.651961] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 5379.721229] Key type lgssc unregistered [ 5379.857521] LNet: 283473:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5379.859804] LNetError: 283473:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5379.870459] LNet: Removed LNI 192.168.203.103@tcp [ 5380.206168] Key type .llcrypt unregistered [ 5380.207415] Key type ._llcrypt unregistered [ 5387.283121] Key type ._llcrypt registered [ 5387.284234] Key type .llcrypt registered [ 5387.329299] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 5387.719686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5387.754520] alg: No test for adler32 (adler32-zlib) [ 5388.614087] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 5388.700279] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 5390.279171] Key type lgssc registered [ 5390.643368] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5393.150385] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5393.156106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5394.322796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5395.671427] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5398.517836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5398.636172] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5399.890960] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5401.608044] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5402.777421] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5404.995471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5405.105911] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5406.123819] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 5406.949865] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5408.670200] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5413.346154] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5413.346516] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5413.350232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5413.355473] Lustre: Skipped 1 previous similar message [ 5415.022652] LustreError: 287081:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5415.060558] Lustre: server umount lustre-OST0000 complete [ 5416.327601] LustreError: 287281:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5416.329215] LustreError: 287281:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5416.351525] Lustre: server umount lustre-MDT0000 complete [ 5417.588535] LustreError: 286309:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763086033 with bad export cookie 12014156274802768083 [ 5417.592553] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5417.593241] LustreError: 286309:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5419.710283] LustreError: 287482:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5419.712713] LustreError: 287482:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5419.780312] Lustre: server umount lustre-MDT0001 complete [ 5423.382200] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 5424.336508] Key type lgssc unregistered [ 5424.468396] LNet: 288013:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5424.470562] LNetError: 288013:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5424.479355] LNet: Removed LNI 192.168.203.103@tcp [ 5424.765891] Key type .llcrypt unregistered [ 5424.766875] Key type ._llcrypt unregistered [ 5427.907976] Key type ._llcrypt registered [ 5427.908866] Key type .llcrypt registered [ 5427.945484] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 5433.057801] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 5438.630645] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 5438.919384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5438.957659] alg: No test for adler32 (adler32-zlib) [ 5439.801666] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 5439.880398] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 5441.455128] Key type lgssc registered [ 5441.805233] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5444.459931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5446.368557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5448.171804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5450.009401] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5452.129111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5452.147051] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5452.151455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5453.238733] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5453.249891] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5453.291284] Lustre: lustre-MDT0000: new disk, initializing [ 5453.320614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5453.326907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5454.613201] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5458.946387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5458.973410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5459.008076] Lustre: 291494:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5459.038307] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5459.041100] Lustre: Skipped 1 previous similar message [ 5459.079071] Lustre: lustre-MDT0001: new disk, initializing [ 5459.117627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5459.131665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5459.137321] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5460.526566] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5463.526506] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5464.671900] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5466.829226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5466.849446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5466.939513] Lustre: lustre-OST0000: new disk, initializing [ 5466.941238] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5466.966710] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5468.893398] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5469.685819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5469.688777] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5469.720075] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5471.848811] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5473.131719] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5473.204476] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5474.385426] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5474.457787] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5479.906950] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5479.907624] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5479.908887] LustreError: Skipped 1 previous similar message [ 5479.915397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5479.917923] Lustre: Skipped 1 previous similar message [ 5481.134097] LustreError: 294432:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5481.169587] Lustre: server umount lustre-OST0000 complete [ 5485.024926] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5485.024970] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5485.031265] Lustre: Skipped 1 previous similar message [ 5485.033201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5485.034783] Lustre: Skipped 1 previous similar message [ 5488.445969] LustreError: 294633:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5488.447701] LustreError: 294633:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5488.477123] Lustre: server umount lustre-MDT0000 complete [ 5489.662153] LustreError: 291485:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763086105 with bad export cookie 16131313188525487075 [ 5489.665205] LustreError: 291485:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5489.666016] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5489.733210] LustreError: 294834:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5489.735178] LustreError: 294834:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5489.784325] Lustre: server umount lustre-MDT0001 complete [ 5492.442031] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 21:08:29 (1763086109) [ 5494.528165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5494.697402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5495.895791] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5498.599411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5498.710282] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5499.850855] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5501.547390] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5502.637694] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5504.749285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5506.493341] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5508.175694] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5528.543877] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5528.546657] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5528.551174] Lustre: Skipped 1 previous similar message [ 5528.552755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5532.397169] LustreError: 297773:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5532.399938] LustreError: 297773:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5532.437916] Lustre: server umount lustre-OST0000 complete [ 5534.688801] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5534.689102] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5534.690749] LustreError: Skipped 1 previous similar message [ 5534.696724] Lustre: Skipped 1 previous similar message [ 5534.698248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5534.701606] Lustre: Skipped 3 previous similar messages [ 5539.808091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5539.809677] Lustre: Skipped 1 previous similar message [ 5539.838384] LustreError: 297975:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5539.840135] LustreError: 297975:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5539.867486] Lustre: server umount lustre-MDT0000 complete [ 5541.076075] LustreError: 297122:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763086157 with bad export cookie 16131313188525487943 [ 5541.078484] LustreError: 296226:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 5541.079117] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5541.081454] LustreError: 297122:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5541.209471] Lustre: server umount lustre-MDT0001 complete [ 5544.694162] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 5545.729830] Key type lgssc unregistered [ 5545.851431] LNet: 298706:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5545.855305] LNetError: 298706:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5545.864362] LNet: Removed LNI 192.168.203.103@tcp [ 5546.167622] Key type .llcrypt unregistered [ 5546.168799] Key type ._llcrypt unregistered [ 5554.427858] Key type ._llcrypt registered [ 5554.428842] Key type .llcrypt registered [ 5554.462595] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 5560.308864] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 5560.651611] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5560.690815] alg: No test for adler32 (adler32-zlib) [ 5561.541599] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 5561.615310] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 5563.191155] Key type lgssc registered [ 5563.542117] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5566.202692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5568.156985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5570.113831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5572.116310] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5573.730459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5575.485256] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5577.753224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5577.773195] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5577.777145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5578.860869] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5578.871713] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5578.907855] Lustre: lustre-MDT0000: new disk, initializing [ 5578.932545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5578.940456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5580.201771] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5584.295797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5584.321715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5584.356709] Lustre: 302481:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5584.377308] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5584.378941] Lustre: Skipped 1 previous similar message [ 5584.412463] Lustre: lustre-MDT0001: new disk, initializing [ 5584.445450] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5584.456854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5584.460881] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5585.760515] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5588.826104] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5590.070953] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5592.455461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5592.477558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5592.573379] Lustre: lustre-OST2710: new disk, initializing [ 5592.575155] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 5592.603957] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 5593.334669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 5593.337745] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 5593.378602] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 5594.426947] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5597.424946] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 5599.672155] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5599.698663] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5599.764707] Lustre: lustre-OST03e8: new disk, initializing [ 5599.766635] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 5599.803712] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 5601.601416] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5602.304356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 5602.309062] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 5602.333984] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 5603.262510] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 5605.805112] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 5605.895138] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5607.162177] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 5607.239452] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5608.567109] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 5608.645147] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5609.930331] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 5610.000667] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5612.515486] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5612.516495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5612.518510] Lustre: Skipped 3 previous similar messages [ 5612.521174] Lustre: Skipped 2 previous similar messages [ 5617.088477] LustreError: 306827:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5617.128239] Lustre: server umount lustre-MDT0000 complete [ 5618.387241] LustreError: 304229:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763086234 with bad export cookie 355562613353028757 [ 5618.391450] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5618.393912] LustreError: 304229:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5618.463849] LustreError: 307028:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5618.465804] LustreError: 307028:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5618.520668] Lustre: server umount lustre-MDT0001 complete [ 5629.938088] LustreError: 307229:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5629.940390] LustreError: 307229:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5629.959108] Lustre: server umount lustre-OST2710 complete [ 5641.394872] LustreError: 307430:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5641.398204] LustreError: 307430:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5641.445865] Lustre: server umount lustre-OST03e8 complete [ 5647.334551] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 5650.485599] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 5653.589619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5655.670184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5657.617233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5659.742233] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5661.985503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5662.006044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5662.085311] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5662.094649] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5662.128862] Lustre: lustre-MDT0000: new disk, initializing [ 5662.151850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5662.158725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5663.413400] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5667.549102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5667.575167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5667.605856] Lustre: 310221:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5667.707268] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5668.973086] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5672.015130] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5673.193463] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5675.396091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5675.419955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5675.510054] Lustre: lustre-OST0000: new disk, initializing [ 5675.511317] Lustre: Skipped 1 previous similar message [ 5675.512911] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5675.514964] Lustre: Skipped 2 previous similar messages [ 5675.543969] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5675.545694] Lustre: Skipped 1 previous similar message [ 5676.154742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5676.157770] Lustre: Skipped 1 previous similar message [ 5676.160335] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5676.198585] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5677.394154] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5680.368267] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5681.717581] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5681.789904] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5683.041779] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5683.109189] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5685.741435] LustreError: 313157:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5685.743100] LustreError: 313157:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5685.779463] Lustre: server umount lustre-OST0000 complete [ 5688.287892] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5688.291644] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5688.295828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5688.297704] Lustre: Skipped 1 previous similar message [ 5691.872324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5691.872817] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5696.992986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5696.994640] Lustre: Skipped 2 previous similar messages [ 5696.997287] LustreError: 313358:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5697.000470] LustreError: 313358:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5697.023098] Lustre: server umount lustre-MDT0000 complete [ 5698.199568] LustreError: 311963:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763086314 with bad export cookie 355562613353030430 [ 5698.202216] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5698.203090] LustreError: 311963:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5708.808948] Lustre: server umount lustre-MDT0001 complete [ 5711.564211] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 21:12:08 (1763086328) [ 5712.061067] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 5712.580727] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 21:12:09 (1763086329) [ 5713.430226] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 5715.880733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5717.194463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5718.398938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5719.591199] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5719.936781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5722.104035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5722.195745] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5722.205428] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5722.272582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5723.477573] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5726.160367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5726.196823] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5726.205708] Lustre: 315688:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5727.602508] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5729.456652] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5730.622755] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5732.890535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5732.919734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5732.993963] Lustre: Denying initial registration attempt from nid 192.168.203.103@tcp, specified as failover [ 5732.997888] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 5733.001692] LustreError: 317303:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -99 [ 5733.004522] LustreError: 317303:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 5733.006432] LustreError: 317303:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 5733.029581] Lustre: server umount lustre-OST0000 complete [ 5733.030902] LustreError: 317303:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 5736.415949] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5736.418097] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5736.421896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5741.377353] LustreError: 317671:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 5741.379478] LustreError: 317671:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5741.406906] Lustre: server umount lustre-MDT0000 complete [ 5742.701686] LustreError: 315680:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763086358 with bad export cookie 355562613353031298 [ 5742.704541] LustreError: 315680:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5742.706256] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5742.835418] Lustre: server umount lustre-MDT0001 complete [ 5746.373756] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 5747.410922] Key type lgssc unregistered [ 5747.529336] LNet: 318403:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5747.532692] LNetError: 318403:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5747.539284] LNet: Removed LNI 192.168.203.103@tcp [ 5747.831945] Key type .llcrypt unregistered [ 5747.832927] Key type ._llcrypt unregistered [ 5751.266564] Key type ._llcrypt registered [ 5751.267554] Key type .llcrypt registered [ 5751.304657] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 5751.634237] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5751.639717] alg: No test for adler32 (adler32-zlib) [ 5752.498557] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 5752.594757] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 5754.175150] Key type lgssc registered [ 5754.531849] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5756.804939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5758.063961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5759.303495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5760.568155] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5760.919574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5767.307161] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 5770.075466] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5770.081576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5771.170628] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5771.178602] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5771.235590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5772.474267] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5775.197537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5775.225685] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5775.236035] Lustre: 321134:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5775.327671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5776.585439] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5778.308191] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5779.519269] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5781.701875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5781.727942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5781.802476] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5781.865214] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5783.725597] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5785.404986] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5786.062095] LustreError: 323414:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5786.095661] Lustre: server umount lustre-OST0000 complete [ 5788.144390] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5788.145179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5788.149391] Lustre: Skipped 1 previous similar message [ 5788.152372] Lustre: Skipped 1 previous similar message [ 5793.248141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5793.249879] Lustre: Skipped 1 previous similar message [ 5793.408068] LustreError: 323614:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 5793.409649] LustreError: 323614:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5793.432833] Lustre: server umount lustre-MDT0000 complete [ 5794.598346] LustreError: 321866:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763086410 with bad export cookie 1155202065841563831 [ 5794.602371] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5794.602942] LustreError: 321866:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5794.673477] LustreError: 323819:0:(obd_class.h:479:obd_check_dev()) Device 7 not setup [ 5794.675295] LustreError: 323819:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5794.726369] Lustre: server umount lustre-MDT0001 complete [ 5798.226447] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 5799.258931] Key type lgssc unregistered [ 5799.384329] LNet: 324350:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5799.388162] LNetError: 324350:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5799.395288] LNet: Removed LNI 192.168.203.103@tcp [ 5799.673487] Key type .llcrypt unregistered [ 5799.675466] Key type ._llcrypt unregistered [ 5808.308803] Key type ._llcrypt registered [ 5808.310707] Key type .llcrypt registered [ 5808.347605] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 5808.745112] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5808.753561] alg: No test for adler32 (adler32-zlib) [ 5809.594287] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 5809.676760] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 5811.255104] Key type lgssc registered [ 5811.570985] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5813.992840] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5814.001516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5815.141717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5816.363041] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5819.117367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5819.236514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5820.461095] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5822.097527] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5823.226165] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5825.384364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5825.493884] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5827.224356] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5828.854319] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5839.840669] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5839.842950] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5839.844037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5839.850046] Lustre: Skipped 2 previous similar messages [ 5840.555878] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.3@tcp (stopping) [ 5840.557656] Lustre: Skipped 2 previous similar messages [ 5843.195266] LustreError: 327972:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5843.223379] Lustre: server umount lustre-MDT0000 complete [ 5844.393973] LustreError: 325707:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763086460 with bad export cookie 6816506822830181335 [ 5844.398376] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5844.960248] LustreError: 325720:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5844.960580] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5844.964595] LustreError: 325720:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5844.968682] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5845.666969] LustreError: 325720:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5850.079732] LustreError: 325720:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5850.080115] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5850.085140] Lustre: Skipped 1 previous similar message [ 5850.491265] LustreError: 328175:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5850.493226] LustreError: 328175:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5850.535460] Lustre: server umount lustre-MDT0001 complete [ 5852.469044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5855.102749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5869.535687] LustreError: 325034:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000075632943 x1848730111202560/t0(0) o250->MGC192.168.203.103@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5869.651247] LustreError: 327331:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5869.674495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5869.692982] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 5870.849841] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5873.577730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5873.710319] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5874.727733] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5875.068331] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5876.795681] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5877.319634] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5878.537892] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5878.820099] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5886.174569] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 7 sec [ 5888.995086] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5888.997061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5888.998276] Lustre: Skipped 1 previous similar message [ 5888.998559] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5889.000543] Lustre: Skipped 1 previous similar message [ 5893.489092] LustreError: 330480:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5893.490921] LustreError: 330480:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5893.531937] Lustre: server umount lustre-OST0000 complete [ 5899.234164] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5899.234338] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5899.234962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5899.236943] Lustre: Skipped 1 previous similar message [ 5900.925198] LustreError: 330681:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5900.926932] LustreError: 330681:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5900.951878] Lustre: server umount lustre-MDT0000 complete [ 5902.191263] LustreError: 327343:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763086518 with bad export cookie 6816506822830188489 [ 5902.194343] LustreError: 327343:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5902.196526] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5902.318281] Lustre: server umount lustre-MDT0001 complete [ 5906.074048] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 5907.056304] Key type lgssc unregistered [ 5907.169331] LNet: 331414:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5907.171351] LNetError: 331414:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5907.178232] LNet: Removed LNI 192.168.203.103@tcp [ 5907.443800] Key type .llcrypt unregistered [ 5907.444764] Key type ._llcrypt unregistered [ 5915.958275] Key type ._llcrypt registered [ 5915.959278] Key type .llcrypt registered [ 5915.997311] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 5916.509931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5916.516456] alg: No test for adler32 (adler32-zlib) [ 5917.358464] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 5917.446832] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 5919.031140] Key type lgssc registered [ 5919.374509] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5921.782728] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5921.788767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5922.955908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5924.216568] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5926.916076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5927.032299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5928.189389] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5929.796761] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5930.872837] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5931.819579] LustreError: 332817:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5931.822021] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5931.822516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5931.824744] LustreError: 332817:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5931.827569] Lustre: Skipped 1 previous similar message [ 5931.836468] LustreError: 332817:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 5936.928501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5936.931843] Lustre: Skipped 2 previous similar messages [ 5942.240055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5942.241689] Lustre: Skipped 1 previous similar message [ 5946.335144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5946.384717] LustreError: 334178:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5946.416947] Lustre: server umount lustre-MDT0000 complete [ 5947.361235] LustreError: 332788:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5947.364881] LustreError: 332788:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5947.602110] LustreError: 334180:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763086563 with bad export cookie 15875995611953288713 [ 5947.604726] LustreError: 334180:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5947.604948] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5947.672153] LustreError: 334383:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5947.673968] LustreError: 334383:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5947.718740] Lustre: server umount lustre-MDT0001 complete [ 5952.062874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5952.145527] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5952.154514] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5952.219427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5953.365383] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5955.991131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5956.021385] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5956.032274] Lustre: 335152:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5956.122912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5957.332710] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5958.974160] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5960.084147] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5962.538434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5962.605606] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 5962.608558] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 5962.610881] LustreError: 336811:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 5962.612803] LustreError: 336811:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 5962.614460] LustreError: 336811:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 5962.632987] Lustre: server umount lustre-OST0000 complete [ 5962.634044] LustreError: 336811:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5965.219900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5965.282392] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5965.343148] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5967.053779] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5967.564028] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 5968.719268] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5971.236312] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5971.263340] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5971.286344] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5971.319711] Lustre: lustre-OST0001: new disk, initializing [ 5971.321396] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5973.154561] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 5973.319522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5973.322448] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5973.362226] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5974.870610] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5978.594567] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5978.595949] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5978.598427] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5978.599396] LustreError: Skipped 1 previous similar message [ 5978.601982] Lustre: Skipped 1 previous similar message [ 5981.871084] LustreError: 338929:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5981.873910] LustreError: 338929:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5981.894932] Lustre: server umount lustre-OST0001 complete [ 5983.200660] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5983.200825] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5983.202653] LustreError: Skipped 1 previous similar message [ 5983.205856] Lustre: Skipped 1 previous similar message [ 5983.207443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5983.210080] Lustre: Skipped 1 previous similar message [ 5983.716121] LustreError: 337728:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5983.723931] LustreError: 337728:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5988.839389] LustreError: 337728:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5988.843229] LustreError: 337728:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5989.228691] LustreError: 339130:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5989.230279] LustreError: 339130:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5989.272977] Lustre: server umount lustre-OST0000 complete [ 5991.904287] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5991.907700] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5991.911620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5991.913237] Lustre: Skipped 2 previous similar messages [ 5996.605721] LustreError: 339332:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5996.607387] LustreError: 339332:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5996.634826] Lustre: server umount lustre-MDT0000 complete [ 5997.838296] LustreError: 335145:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763086614 with bad export cookie 15875995611953289231 [ 5997.841378] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5997.841601] LustreError: 335145:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5997.973793] Lustre: server umount lustre-MDT0001 complete [ 6001.402593] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 6002.426753] Key type lgssc unregistered [ 6002.557402] LNet: 340065:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6002.560829] LNetError: 340065:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6002.571454] LNet: Removed LNI 192.168.203.103@tcp [ 6002.867164] Key type .llcrypt unregistered [ 6002.868243] Key type ._llcrypt unregistered [ 6013.209808] Key type ._llcrypt registered [ 6013.210747] Key type .llcrypt registered [ 6013.244037] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 6019.257840] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 6019.637294] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6019.692404] alg: No test for adler32 (adler32-zlib) [ 6020.549256] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 6020.636121] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 6022.215110] Key type lgssc registered [ 6022.570845] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6025.406568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6027.536129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6029.424674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6031.512316] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6033.702082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6033.722262] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6033.726634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6034.821542] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6034.837793] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6034.880094] Lustre: lustre-MDT0000: new disk, initializing [ 6034.906297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6034.914891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6036.231153] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 6040.396598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6040.416222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6040.457354] Lustre: 343986:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6040.484294] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6040.486055] Lustre: Skipped 1 previous similar message [ 6040.521116] Lustre: lustre-MDT0001: new disk, initializing [ 6040.560650] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6040.575475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6040.580543] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6042.029495] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 6045.243919] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6046.399626] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6048.592046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6048.622528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6048.732455] Lustre: lustre-OST0000: new disk, initializing [ 6048.735353] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6048.770537] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6050.633410] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 6053.740852] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6055.211312] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6055.426489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6055.429348] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6055.466880] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6056.319491] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 6057.639699] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6057.717393] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6060.514186] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6060.515682] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6060.516173] LustreError: Skipped 1 previous similar message [ 6060.520313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6060.522695] Lustre: Skipped 1 previous similar message [ 6064.428083] LustreError: 346932:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6064.464120] Lustre: server umount lustre-OST0000 complete [ 6066.143925] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6066.146522] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6066.151156] Lustre: Skipped 1 previous similar message [ 6066.152985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6070.752544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6070.752867] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6071.806054] LustreError: 347133:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6071.807988] LustreError: 347133:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6071.833929] Lustre: server umount lustre-MDT0000 complete [ 6073.030284] LustreError: 343979:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763086689 with bad export cookie 16892108406091633858 [ 6073.033802] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6073.035884] LustreError: 343979:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6073.106113] LustreError: 347334:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6073.107983] LustreError: 347334:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6073.155077] Lustre: server umount lustre-MDT0001 complete [ 6075.874710] Lustre: DEBUG MARKER: == conf-sanity test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 21:18:13 (1763086693) [ 6077.749675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6079.660843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6086.846351] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 6089.688723] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 6092.919372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6094.941905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6096.822711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6098.732953] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6100.852159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6100.870953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6100.948273] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6100.958328] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6100.995595] Lustre: lustre-MDT0000: new disk, initializing [ 6101.021326] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6101.027162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6102.259469] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 6106.362741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6106.388524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6106.420370] Lustre: 351101:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6106.447145] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6106.449568] Lustre: Skipped 1 previous similar message [ 6106.485847] Lustre: lustre-MDT0001: new disk, initializing [ 6106.516899] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6106.527764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6106.530621] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6107.783387] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 6110.919619] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6112.010962] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6114.134640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6114.160492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6114.545109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6114.550027] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6114.577744] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6116.055874] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 6119.018744] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6120.303106] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6120.374637] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6121.588952] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6121.666290] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6125.026861] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6125.027079] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6125.029215] LustreError: Skipped 1 previous similar message [ 6125.032538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6125.034669] Lustre: Skipped 2 previous similar messages [ 6128.364159] LustreError: 354039:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6128.365858] LustreError: 354039:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6128.396259] Lustre: server umount lustre-OST0000 complete [ 6130.144805] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6130.145551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6130.149739] Lustre: Skipped 2 previous similar messages [ 6135.677594] LustreError: 354240:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6135.681888] LustreError: 354240:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6135.713863] Lustre: server umount lustre-MDT0000 complete [ 6136.866972] LustreError: 351094:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763086753 with bad export cookie 16892108406091634712 [ 6136.869830] LustreError: 351094:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6136.870649] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6136.977740] Lustre: server umount lustre-MDT0001 complete [ 6139.710489] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 21:19:16 (1763086756) [ 6142.124118] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 21:19:19 (1763086759) [ 6144.814827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6144.979456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6144.982508] Lustre: Skipped 1 previous similar message [ 6146.163410] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 6148.783252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6150.069225] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 6151.623810] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6152.779215] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6154.893556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6156.683972] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 6158.387416] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6162.620353] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 6163.095179] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 6163.543117] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 6164.015531] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 6164.572363] Lustre: Failing over lustre-MDT0000 [ 6164.648447] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.3@tcp (stopping) [ 6164.650304] Lustre: Skipped 3 previous similar messages [ 6164.705294] LustreError: 357893:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6164.706996] LustreError: 357893:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6164.738972] Lustre: server umount lustre-MDT0000 complete [ 6169.571496] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6169.572735] LustreError: 355454:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6169.572758] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6169.576627] Lustre: Skipped 2 previous similar messages [ 6169.579854] LustreError: 355454:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6174.689793] LustreError: 355454:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6174.696638] LustreError: 355454:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6177.517446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6177.560844] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6177.654435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6177.656110] Lustre: Skipped 2 previous similar messages [ 6177.682322] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6178.925540] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 6180.002636] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6182.888607] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6182.894909] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 6182.915896] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6183.970193] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6184.492840] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6186.265206] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 6186.882586] Lustre: Failing over lustre-MDT0000 [ 6187.091079] LustreError: 358971:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6187.092878] LustreError: 358971:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6187.119118] Lustre: server umount lustre-MDT0000 complete [ 6188.002355] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6188.003620] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6188.005569] LustreError: 355454:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6188.005577] LustreError: 355454:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6188.015343] Lustre: Skipped 2 previous similar messages [ 6190.248650] LustreError: 357239:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6190.252725] LustreError: 357239:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6195.363394] LustreError: 355448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6195.367921] LustreError: 355448:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6199.856798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6199.894973] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6200.012423] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6200.482368] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6201.199145] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 6205.416111] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6205.417849] Lustre: Skipped 2 previous similar messages [ 6205.421936] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 6205.440412] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6206.567424] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6207.071654] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6210.532353] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6210.533599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6210.537579] Lustre: Skipped 2 previous similar messages [ 6215.394843] Lustre: server umount lustre-MDT0000 complete [ 6215.649462] LustreError: 355454:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6215.653307] LustreError: 355454:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6216.607567] LustreError: 355478:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763086832 with bad export cookie 16892108406091637379 [ 6216.614109] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6216.736260] Lustre: server umount lustre-MDT0001 complete [ 6228.272789] LustreError: 360403:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6228.275368] LustreError: 360403:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 6228.310372] Lustre: server umount lustre-OST0000 complete [ 6233.134621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6233.288237] LustreError: 361231:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6233.311545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6233.313282] Lustre: Skipped 1 previous similar message [ 6234.546180] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 6237.104749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6238.340792] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 6239.953221] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6241.028662] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6243.117201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6244.264607] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6244.944392] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 6246.513939] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6254.559905] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6254.563027] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6254.567390] Lustre: Skipped 3 previous similar messages [ 6254.569821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6258.704393] Lustre: server umount lustre-OST0000 complete [ 6266.005261] Lustre: server umount lustre-MDT0000 complete [ 6267.211476] LustreError: 361210:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763086883 with bad export cookie 16892108406091638100 [ 6267.214856] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6267.217276] LustreError: 361210:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 6267.339142] Lustre: server umount lustre-MDT0001 complete [ 6270.764225] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 6271.722164] Key type lgssc unregistered [ 6271.844350] LNet: 364412:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6271.847284] LNetError: 364412:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6271.860345] LNet: Removed LNI 192.168.203.103@tcp [ 6272.143262] Key type .llcrypt unregistered [ 6272.144287] Key type ._llcrypt unregistered [ 6281.260735] Key type ._llcrypt registered [ 6281.262196] Key type .llcrypt registered [ 6281.299098] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 6281.695692] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6281.707515] alg: No test for adler32 (adler32-zlib) [ 6282.564478] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 6282.638159] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 6284.215107] Key type lgssc registered [ 6284.542574] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6286.929612] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6286.934960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6288.076930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6289.248032] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 6291.981493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6292.104305] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6293.287730] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 6294.928874] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6296.041057] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6298.190700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6298.290717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6299.302507] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6299.989749] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 6301.582134] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6307.810282] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6307.812086] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6307.813162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6307.815739] Lustre: Skipped 1 previous similar message [ 6312.318933] LustreError: 368126:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6312.349864] Lustre: server umount lustre-MDT0000 complete [ 6312.930364] LustreError: 365882:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6312.934838] LustreError: 365882:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6313.483098] LustreError: 365865:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763086929 with bad export cookie 11288129747218037991 [ 6313.485878] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6313.486257] LustreError: 365865:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6313.551601] LustreError: 368331:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6313.553416] LustreError: 368331:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6313.592929] Lustre: server umount lustre-MDT0001 complete [ 6324.979796] LustreError: 368532:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6324.981412] LustreError: 368532:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6325.021627] Lustre: server umount lustre-OST0000 complete [ 6329.466322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6329.630185] LustreError: 369261:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6329.653884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6330.916967] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 6333.642443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6333.765833] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6334.880556] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 6336.446537] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6337.521439] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6339.553479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6341.338787] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 6343.009237] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6354.023090] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 6369.763601] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6369.765040] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6369.766678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6369.766690] Lustre: Skipped 2 previous similar messages [ 6369.771569] Lustre: Skipped 3 previous similar messages [ 6371.469797] LustreError: 371630:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6371.474144] LustreError: 371630:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6371.514709] Lustre: server umount lustre-MDT0000 complete [ 6373.219439] LustreError: 369241:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763086989 with bad export cookie 11288129747218039076 [ 6373.224117] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6373.226290] LustreError: 369241:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6373.367628] Lustre: server umount lustre-MDT0001 complete [ 6385.209354] LustreError: 372033:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6385.215215] LustreError: 372033:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 6385.260768] Lustre: server umount lustre-OST0000 complete [ 6391.892200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6392.051700] LustreError: 372861:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6392.073902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6392.077298] Lustre: Skipped 1 previous similar message [ 6393.234001] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 6397.246403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6398.927222] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 6401.097459] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6402.692205] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6405.838449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6408.326601] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 6410.868819] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6417.257676] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 6427.616558] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6427.616914] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6427.618633] LustreError: Skipped 1 previous similar message [ 6427.622424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6427.624892] Lustre: Skipped 3 previous similar messages [ 6432.738019] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6432.739491] Lustre: Skipped 1 previous similar message [ 6436.831101] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6436.850108] LustreError: 375128:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6436.853357] LustreError: 375128:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6436.894621] Lustre: server umount lustre-OST0000 complete [ 6438.367690] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6438.369517] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6438.372347] Lustre: Skipped 1 previous similar message [ 6438.373729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6442.978423] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6444.127079] Lustre: server umount lustre-MDT0000 complete [ 6445.658223] LustreError: 372841:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763087061 with bad export cookie 11288129747218040084 [ 6445.665224] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6445.668417] LustreError: 372841:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6445.802668] Lustre: server umount lustre-MDT0001 complete [ 6450.319365] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 6451.467426] Key type lgssc unregistered [ 6451.623354] LNet: 376065:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6451.625444] LNetError: 376065:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6451.636330] LNet: Removed LNI 192.168.203.103@tcp [ 6452.052079] Key type .llcrypt unregistered [ 6452.054257] Key type ._llcrypt unregistered [ 6462.022642] Key type ._llcrypt registered [ 6462.023767] Key type .llcrypt registered [ 6462.066556] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 6462.477199] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6462.488373] alg: No test for adler32 (adler32-zlib) [ 6463.347505] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 6463.434301] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 6465.023169] Key type lgssc registered [ 6465.462338] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6468.894282] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6468.905430] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6468.909636] LustreError: 377462:0:(osd_handler.c:8577:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 6468.919725] LustreError: 377462:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 6468.922948] LustreError: 377462:0:(obd_mount.c:193:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 6468.926380] LustreError: 377462:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 6468.930235] LustreError: 377462:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6472.515270] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6472.520504] LustreError: 377860:0:(osd_handler.c:8577:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 6472.528791] LustreError: 377860:0:(obd_config.c:783:class_setup()) setup lustre-OST0000-osd failed (-22) [ 6472.533352] LustreError: 377860:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 6472.537571] LustreError: 377860:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 6472.542417] LustreError: 377860:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6480.187419] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 6481.577245] Key type lgssc unregistered [ 6481.727661] LNet: 378844:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6481.731951] LNetError: 378844:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6481.744483] LNet: Removed LNI 192.168.203.103@tcp [ 6482.204160] Key type .llcrypt unregistered [ 6482.206410] Key type ._llcrypt unregistered [ 6489.693612] Key type ._llcrypt registered [ 6489.694547] Key type .llcrypt registered [ 6489.735400] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 6496.900560] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 6497.226795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6497.260112] alg: No test for adler32 (adler32-zlib) [ 6498.130777] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 6498.208195] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 6499.783157] Key type lgssc registered [ 6500.336786] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6503.819213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6506.519734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6508.859862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6511.275344] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6513.726869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6513.764589] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6513.773118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6514.867567] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6514.879638] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6514.915159] Lustre: lustre-MDT0000: new disk, initializing [ 6514.943123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6514.950409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6516.616283] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 6521.846849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6521.877418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6521.927633] Lustre: 381947:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6521.959824] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6521.962855] Lustre: Skipped 1 previous similar message [ 6521.996778] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6522.016502] Lustre: lustre-MDT0001: new disk, initializing [ 6522.058142] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6522.076668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6522.082536] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6523.414529] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 6526.906315] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6528.440670] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6530.953838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6530.992246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6531.135572] Lustre: lustre-OST0000: new disk, initializing [ 6531.138171] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6531.177145] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6533.408757] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 6534.103451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6534.109963] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6534.164949] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6536.890553] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6538.584650] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6538.660783] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6540.104869] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6540.241972] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6544.353841] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6544.358406] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6544.359428] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6544.365298] Lustre: Skipped 1 previous similar message [ 6547.246799] LustreError: 384885:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6547.278210] Lustre: server umount lustre-OST0000 complete [ 6549.472723] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6549.473403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6549.475879] Lustre: Skipped 1 previous similar message [ 6549.477287] Lustre: Skipped 1 previous similar message [ 6554.591958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6554.593872] Lustre: Skipped 2 previous similar messages [ 6554.625658] LustreError: 385087:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6554.627446] LustreError: 385087:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6554.655229] Lustre: server umount lustre-MDT0000 complete [ 6555.796140] LustreError: 381940:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763087172 with bad export cookie 2871471033449222857 [ 6555.798871] LustreError: 381940:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6555.799979] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6555.870444] LustreError: 385288:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6555.872059] LustreError: 385288:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6555.917719] Lustre: server umount lustre-MDT0001 complete [ 6558.517288] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 6558.989314] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 21:26:16 (1763087176) [ 6560.977911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6561.132329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6562.228199] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 6564.732207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6566.055345] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 6567.649807] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6568.669405] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6570.689366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6570.812756] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6570.816610] Lustre: Skipped 1 previous similar message [ 6572.546044] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 6574.101915] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6576.378573] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6576.405851] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6576.481691] Lustre: lustre-OST0001: new disk, initializing [ 6576.483247] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6577.679626] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6577.683316] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6577.728562] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6578.513343] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 6581.498388] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6587.873498] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6587.877744] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6587.879126] LustreError: Skipped 2 previous similar messages [ 6587.880188] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6587.885153] Lustre: Skipped 1 previous similar message [ 6589.358212] LustreError: 389205:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 6589.360301] LustreError: 389205:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6589.379722] Lustre: server umount lustre-OST0001 complete [ 6592.996924] LustreError: 388707:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6593.004319] LustreError: 388707:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6598.121668] LustreError: 387605:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6598.127029] LustreError: 387605:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6602.720051] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6602.722609] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6602.725915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6602.727525] Lustre: Skipped 2 previous similar messages [ 6603.238461] LustreError: 389207:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6603.242758] LustreError: 389207:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6607.663125] LustreError: 389431:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6607.666983] LustreError: 389431:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6607.705286] Lustre: server umount lustre-OST0000 complete [ 6610.912123] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6610.916909] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6610.923618] Lustre: Skipped 1 previous similar message [ 6615.126841] Lustre: server umount lustre-MDT0000 complete [ 6616.395779] LustreError: 385986:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763087232 with bad export cookie 2871471033449223739 [ 6616.400695] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6616.401351] LustreError: 385986:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6622.537468] LustreError: 389833:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 6622.540011] LustreError: 389833:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 6622.599841] Lustre: server umount lustre-MDT0001 complete [ 6625.979713] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 6627.326385] Key type lgssc unregistered [ 6627.484665] LNet: 390365:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6627.489059] LNetError: 390365:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6627.496735] LNet: Removed LNI 192.168.203.103@tcp [ 6627.905828] Key type .llcrypt unregistered [ 6627.907631] Key type ._llcrypt unregistered [ 6639.949110] Key type ._llcrypt registered [ 6639.950343] Key type .llcrypt registered [ 6639.985679] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 6646.008261] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 6646.418539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6646.434357] alg: No test for adler32 (adler32-zlib) [ 6647.287831] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 6647.372710] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 6648.975200] Key type lgssc registered [ 6649.454803] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6652.185115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6654.562783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6657.007500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6659.890609] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6663.337750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6663.369972] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6663.378031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6664.474987] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6664.486625] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6664.522450] Lustre: lustre-MDT0000: new disk, initializing [ 6664.548776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6664.557462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6666.324493] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 6671.678344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6671.706812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6671.752106] Lustre: 394286:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6671.784156] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6671.788020] Lustre: Skipped 1 previous similar message [ 6671.841356] Lustre: lustre-MDT0001: new disk, initializing [ 6671.873789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6671.886296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6671.890303] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6673.528942] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 6677.602731] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6679.360787] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6682.660344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6682.696416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6682.834681] Lustre: lustre-OST0000: new disk, initializing [ 6682.837814] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6682.881355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6685.253109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6685.256839] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6685.304019] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6685.314638] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 6689.267965] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6690.963260] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6691.062414] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6692.764340] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6692.876232] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6695.907493] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6695.908289] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6695.911891] LustreError: Skipped 1 previous similar message [ 6695.921451] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6695.926530] Lustre: Skipped 1 previous similar message [ 6701.027113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6701.030389] Lustre: Skipped 1 previous similar message [ 6706.148355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6706.154866] Lustre: Skipped 1 previous similar message [ 6708.191104] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6708.212292] LustreError: 397227:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6708.268451] Lustre: server umount lustre-OST0000 complete [ 6711.265869] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6711.267120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6711.272157] Lustre: Skipped 2 previous similar messages [ 6716.117097] LustreError: 397430:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6716.121139] LustreError: 397430:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6716.160131] Lustre: server umount lustre-MDT0000 complete [ 6717.585130] LustreError: 395068:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763087333 with bad export cookie 12743678601450011786 [ 6717.587968] LustreError: 395068:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6717.588639] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6717.654403] LustreError: 397632:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6717.656056] LustreError: 397632:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6717.701346] Lustre: server umount lustre-MDT0001 complete [ 6721.160143] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 21:28:58 (1763087338) [ 6725.499215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6729.028612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6729.166946] Lustre: lustre-MDT0000: new disk, initializing [ 6729.176582] LustreError: 398898:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6729.182190] LustreError: 398898:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6729.197412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6730.369774] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 6732.952262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6734.168160] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 6735.797600] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6736.908229] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6737.551823] LustreError: 398926:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6737.554165] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6737.554586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6737.554590] Lustre: Skipped 1 previous similar message [ 6737.556538] LustreError: 398926:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6737.561836] Lustre: Skipped 1 previous similar message [ 6737.568020] LustreError: 398926:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -5 [ 6743.676288] LustreError: 400240:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6743.677923] LustreError: 400240:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6743.702858] Lustre: server umount lustre-MDT0000 complete [ 6744.034300] LustreError: 399609:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6744.038084] LustreError: 399609:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6744.876065] LustreError: 398877:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763087361 with bad export cookie 12743678601450012668 [ 6744.879465] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6744.881038] LustreError: 398877:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6744.978332] Lustre: server umount lustre-MDT0001 complete [ 6748.487989] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 21:29:25 (1763087365) [ 6751.354926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6751.598651] LustreError: 401161:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6751.629170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6751.633583] Lustre: Skipped 1 previous similar message [ 6753.377745] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 6756.832791] LustreError: 401161:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6757.455928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6759.458728] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 6761.883839] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6763.259496] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6766.160495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6766.301597] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6766.305434] Lustre: Skipped 1 previous similar message [ 6768.914450] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 6771.607766] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6785.483440] Lustre: Permanently deactivating lustre-OST0000 [ 6785.492615] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6792.690994] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6793.320213] LustreError: 403877:0:(mgs_llog.c:1938:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 6793.326119] LustreError: 403877:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [ 6793.329934] LustreError: 403877:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6794.031053] LustreError: 403925:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [ 6794.035677] LustreError: 403925:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6800.946125] LustreError: 404023:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6800.949902] LustreError: 404023:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6801.002653] Lustre: server umount lustre-OST0000 complete [ 6802.259494] LustreError: 404175:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [ 6802.262975] LustreError: 404175:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6803.425194] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6803.425993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6803.436388] Lustre: Skipped 1 previous similar message [ 6803.440216] Lustre: Skipped 3 previous similar messages [ 6813.670946] LustreError: 404272:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6813.674703] LustreError: 404272:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6813.712206] Lustre: server umount lustre-MDT0000 complete [ 6815.540807] LustreError: 401140:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763087431 with bad export cookie 12743678601450013186 [ 6815.544729] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6815.548224] LustreError: 401140:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6825.009553] Lustre: server umount lustre-MDT0001 complete [ 6829.150374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6830.893203] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 6831.789140] LustreError: 405316:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 6834.677559] Lustre: 405654:0:(mgs_llog.c:1699:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.103@tcp [ 6835.593611] LustreError: 405751:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6835.596325] LustreError: 405751:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6835.621221] Lustre: server umount MGS complete [ 6840.365424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6840.588386] LustreError: 406379:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6840.593380] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6840.596232] Lustre: Skipped 1 previous similar message [ 6840.618575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6842.335869] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 6846.363106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6846.501880] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6848.225174] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 6850.626006] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6852.333808] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6853.285348] Lustre: Permanently reactivating lustre-OST0000 [ 6853.296076] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6853.301196] Lustre: Skipped 2 previous similar messages [ 6877.151141] LustreError: 407885:0:(osp_dev.c:692:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 6879.983860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6880.101293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6880.103020] Lustre: Skipped 1 previous similar message [ 6881.926895] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 6883.557590] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6897.639461] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6897.640270] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6897.643961] LustreError: Skipped 1 previous similar message [ 6897.651896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6897.656400] Lustre: Skipped 6 previous similar messages [ 6901.484495] LustreError: 409714:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6901.486559] LustreError: 409714:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 6901.518712] Lustre: server umount lustre-OST0000 complete [ 6902.754158] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6908.826578] Lustre: server umount lustre-MDT0000 complete [ 6910.084315] LustreError: 406359:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763087526 with bad export cookie 12743678601450014131 [ 6910.090653] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6910.092743] LustreError: 406359:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6910.200077] Lustre: server umount lustre-MDT0001 complete [ 6913.576666] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 6914.839294] Key type lgssc unregistered [ 6914.984528] LNet: 410653:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6914.988874] LNetError: 410653:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6914.997695] LNet: Removed LNI 192.168.203.103@tcp [ 6915.398456] Key type .llcrypt unregistered [ 6915.399356] Key type ._llcrypt unregistered [ 6921.639349] Key type ._llcrypt registered [ 6921.640246] Key type .llcrypt registered [ 6921.674580] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 6927.784276] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 6928.206831] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6928.219922] alg: No test for adler32 (adler32-zlib) [ 6929.066930] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 6929.147823] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 6930.719138] Key type lgssc registered [ 6931.035632] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6933.928104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6936.461151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6938.229570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6940.775504] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6943.015699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6943.043983] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6943.051423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6944.143623] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6944.153274] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6944.192996] Lustre: lustre-MDT0000: new disk, initializing [ 6944.218730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6944.225456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6945.469519] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 6949.650920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6949.692679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6949.754316] Lustre: 413757:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6949.794614] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6949.798226] Lustre: Skipped 1 previous similar message [ 6949.851224] Lustre: lustre-MDT0001: new disk, initializing [ 6949.898075] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6949.917761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6949.925129] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6951.288629] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 6954.445217] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6955.748251] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6958.360405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6958.396442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6958.534092] Lustre: lustre-OST0000: new disk, initializing [ 6958.536819] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6958.576897] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6960.976880] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 6961.423372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6961.432118] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6961.478714] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6964.042778] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6965.870754] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6966.002907] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6967.653005] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6967.758519] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6971.876895] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6971.877554] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6971.881392] LustreError: Skipped 1 previous similar message [ 6971.889167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6971.894453] Lustre: Skipped 1 previous similar message [ 6974.705176] LustreError: 416694:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6974.764335] Lustre: server umount lustre-OST0000 complete [ 6976.994233] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6976.995340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6977.001445] Lustre: Skipped 2 previous similar messages [ 6982.113814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6982.117587] Lustre: Skipped 2 previous similar messages [ 6982.674871] LustreError: 416895:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6982.678891] LustreError: 416895:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6982.714072] Lustre: server umount lustre-MDT0000 complete [ 6984.512431] LustreError: 413748:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763087600 with bad export cookie 14633068959954384387 [ 6984.519109] LustreError: 413748:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6984.519223] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6984.617412] LustreError: 417098:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6984.620630] LustreError: 417098:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6984.681052] Lustre: server umount lustre-MDT0001 complete [ 6988.740729] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 6989.578177] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 21:33:26 (1763087606) [ 6992.956154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6993.223506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6994.491088] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 6998.404744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6998.497361] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6998.602923] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7000.065339] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 7001.826400] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7003.415402] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7006.745268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7006.943355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7009.754255] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 7012.294193] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7013.277825] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 7024.417654] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 7028.704744] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7028.709181] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7028.717061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7029.734084] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7032.562180] LustreError: 420275:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7032.566222] LustreError: 420275:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7032.631835] Lustre: server umount lustre-OST0000 complete [ 7034.849617] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7034.858307] Lustre: Skipped 1 previous similar message [ 7039.456685] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7039.460912] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7039.468996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7039.472859] Lustre: Skipped 3 previous similar messages [ 7040.650908] LustreError: 420477:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7040.654216] LustreError: 420477:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7040.702749] Lustre: server umount lustre-MDT0000 complete [ 7042.588394] LustreError: 417797:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763087658 with bad export cookie 14633068959954385255 [ 7042.591117] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7042.592234] LustreError: 417797:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7042.740097] Lustre: server umount lustre-MDT0001 complete [ 7047.231581] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 7048.551057] Key type lgssc unregistered [ 7048.725696] LNet: 421209:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7048.729979] LNetError: 421209:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7048.739650] LNet: Removed LNI 192.168.203.103@tcp [ 7049.240096] Key type .llcrypt unregistered [ 7049.242662] Key type ._llcrypt unregistered [ 7059.141845] Key type ._llcrypt registered [ 7059.143096] Key type .llcrypt registered [ 7059.182767] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 7059.294334] Key type .llcrypt unregistered [ 7059.297560] Key type ._llcrypt unregistered [ 7067.744690] Key type ._llcrypt registered [ 7067.746666] Key type .llcrypt registered [ 7067.806265] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 7068.116600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7068.148048] alg: No test for adler32 (adler32-zlib) [ 7069.038347] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 7069.140680] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 7070.735203] Key type lgssc registered [ 7071.363191] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7075.303390] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7075.314394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7076.541697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7078.302694] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 7081.959323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7082.142700] LustreError: 424086:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7082.148466] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7082.152298] LustreError: 424086:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7084.619879] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 7086.241252] LustreError: 423358:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7086.250694] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7086.251074] LustreError: 423358:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7086.874695] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7090.167678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7090.369459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7091.941522] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 7100.899166] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7100.902106] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7100.902367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7100.905330] Lustre: Skipped 1 previous similar message [ 7105.646162] LustreError: 425415:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 7105.675377] Lustre: server umount lustre-OST0000 complete [ 7110.113366] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 7110.115382] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7110.118189] Lustre: Skipped 1 previous similar message [ 7110.119629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7111.136136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7111.136644] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7116.257159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7116.260781] Lustre: Skipped 2 previous similar messages [ 7121.375180] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7121.377481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7121.388132] Lustre: Skipped 1 previous similar message [ 7121.446037] LustreError: 425616:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7121.449337] LustreError: 425616:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7121.480554] Lustre: server umount lustre-MDT0000 complete [ 7123.236238] LustreError: 425329:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763087739 with bad export cookie 8420901161372504175 [ 7123.245948] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7123.346965] LustreError: 425819:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7123.350712] LustreError: 425819:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7123.410701] Lustre: server umount lustre-MDT0001 complete [ 7127.869479] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 7129.272231] Key type lgssc unregistered [ 7129.443697] LNet: 426350:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7129.448219] LNetError: 426350:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7129.457508] LNet: Removed LNI 192.168.203.103@tcp [ 7129.875593] Key type .llcrypt unregistered [ 7129.877656] Key type ._llcrypt unregistered [ 7138.949452] Key type ._llcrypt registered [ 7138.952188] Key type .llcrypt registered [ 7138.998779] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 7139.299188] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7139.331613] alg: No test for adler32 (adler32-zlib) [ 7140.184346] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 7140.265462] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 7141.855141] Key type lgssc registered [ 7142.360615] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7146.214205] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7146.225847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7161.695380] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7166.815275] LustreError: 427698:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.103@tcp: failed processing log, type 4: rc = -110 [ 7193.439236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7195.956588] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 7198.018181] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7200.674572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7200.940503] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7200.974956] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 7202.546928] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 7205.219815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7205.342289] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7206.644196] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 7208.973670] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7210.749124] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7213.540799] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7213.543074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7213.550841] Lustre: Skipped 1 previous similar message [ 7218.659861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7218.664490] Lustre: Skipped 2 previous similar messages [ 7219.379872] LustreError: 429980:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 7219.445508] Lustre: server umount lustre-OST0000 complete [ 7223.780208] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7223.783356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7223.788278] Lustre: Skipped 1 previous similar message [ 7227.474643] LustreError: 430181:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 7227.478587] LustreError: 430181:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7227.526800] Lustre: server umount lustre-MDT0000 complete [ 7228.897060] LustreError: 428626:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7228.903297] LustreError: 428626:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7229.287512] LustreError: 427707:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763087845 with bad export cookie 1213382258208704290 [ 7229.291911] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7229.364418] LustreError: 430383:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7229.366112] LustreError: 430383:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7229.420879] Lustre: server umount lustre-MDT0001 complete [ 7234.606924] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 7236.224884] Key type lgssc unregistered [ 7236.417670] LNet: 430915:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7236.422775] LNetError: 430915:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7236.431767] LNet: Removed LNI 192.168.203.103@tcp [ 7236.925607] Key type .llcrypt unregistered [ 7236.929099] Key type ._llcrypt unregistered [ 7246.861818] Key type ._llcrypt registered [ 7246.862748] Key type .llcrypt registered [ 7246.894322] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 7247.264384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7247.273511] alg: No test for adler32 (adler32-zlib) [ 7248.115587] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 7248.190398] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 7249.767143] Key type lgssc registered [ 7250.095603] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7252.419654] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7252.425134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7253.569296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7254.728315] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 7257.362595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7257.479761] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7258.673199] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 7260.304623] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7261.423767] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7263.526173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7263.643628] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7264.682198] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 7265.391682] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 7267.081716] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7272.933174] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7272.935488] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7272.936164] Lustre: Skipped 2 previous similar messages [ 7272.937391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7273.649017] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.3@tcp (stopping) [ 7273.650715] Lustre: Skipped 2 previous similar messages [ 7275.093470] LustreError: 434529:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7275.131606] Lustre: server umount lustre-MDT0000 complete [ 7276.283280] LustreError: 432269:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.3@tcp arrived at 1763087892 with bad export cookie 1959337450453528518 [ 7276.291229] LustreError: 432269:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7277.104145] LustreError: 432269:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763087893 with bad export cookie 1959337450453527671 [ 7277.112374] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7278.049133] LustreError: 432309:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7278.050434] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7278.051111] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7278.052273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7278.058184] LustreError: 432309:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7282.080635] LustreError: 432309:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7282.081368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7282.089818] LustreError: 432309:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7283.186162] LustreError: 434736:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7283.189872] LustreError: 434736:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7283.248160] Lustre: server umount lustre-OST0000 complete [ 7286.243524] LustreError: 435089:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7286.245959] LustreError: 435089:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7286.308767] Lustre: server umount lustre-MDT0001 complete [ 7291.171832] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 7292.668420] Key type lgssc unregistered [ 7292.833601] LNet: 435621:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7292.838449] LNetError: 435621:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7292.851520] LNet: Removed LNI 192.168.203.103@tcp [ 7293.324105] Key type .llcrypt unregistered [ 7293.326039] Key type ._llcrypt unregistered [ 7304.113948] Key type ._llcrypt registered [ 7304.114994] Key type .llcrypt registered [ 7304.155792] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 7304.463363] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7304.528992] alg: No test for adler32 (adler32-zlib) [ 7305.368497] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 7305.444949] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 7307.015159] Key type lgssc registered [ 7307.596684] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7311.358269] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7311.371916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7312.570305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7314.107731] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 7318.139930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7318.304796] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7320.006822] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 7322.211278] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7323.781973] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7326.962769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7327.114674] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7329.755641] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 7332.285492] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7337.388542] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 7342.564841] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7342.573510] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7342.578613] Lustre: Skipped 1 previous similar message [ 7344.590092] LustreError: 439243:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7344.623391] Lustre: server umount lustre-MDT0001 complete [ 7347.681651] LustreError: 436996:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7347.683293] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7347.683767] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7347.683787] Lustre: Skipped 1 previous similar message [ 7347.685841] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7347.685849] Lustre: Skipped 1 previous similar message [ 7347.690475] LustreError: 436996:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7352.801359] LustreError: 436996:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7352.801739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7352.827122] LustreError: 439446:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7352.830997] LustreError: 439446:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7352.879913] Lustre: server umount lustre-OST0000 complete [ 7354.569897] LustreError: 439648:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7354.573256] LustreError: 439648:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7354.643717] Lustre: server umount lustre-MDT0000 complete [ 7360.378600] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 7361.618876] Key type lgssc unregistered [ 7361.736384] LNet: 440333:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7361.738197] LNetError: 440333:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7361.744285] LNet: Removed LNI 192.168.203.103@tcp [ 7362.024274] Key type .llcrypt unregistered [ 7362.025778] Key type ._llcrypt unregistered [ 7372.773947] Key type ._llcrypt registered [ 7372.777164] Key type .llcrypt registered [ 7372.849911] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 7380.297366] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 7380.693306] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7380.701408] alg: No test for adler32 (adler32-zlib) [ 7381.555476] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 7381.652637] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 7383.255109] Key type lgssc registered [ 7383.674731] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7386.220565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7388.128273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7389.928145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7391.936480] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7394.500720] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 7397.079672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7399.020904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7399.037039] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7399.040929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7400.115749] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7400.125309] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7400.160487] Lustre: lustre-MDT0000: new disk, initializing [ 7400.182362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7400.187692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7401.413685] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 7405.051873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7407.387265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7407.408477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7407.510426] Lustre: lustre-OST0000: new disk, initializing [ 7407.512605] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7407.514512] Lustre: Skipped 1 previous similar message [ 7407.515606] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 7407.537429] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7409.251235] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 7412.361146] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7412.722654] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 7412.727972] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 7412.736600] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 7421.034535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7423.205797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7423.231477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7423.278159] Lustre: 444276:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7423.309406] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7423.348483] Lustre: lustre-MDT0001: new disk, initializing [ 7423.389050] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7423.402741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 7423.405319] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 7424.701808] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 7428.180714] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7428.244855] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7429.674239] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7432.838622] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 3 sec [ 7437.492899] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.3@tcp (stopping) [ 7437.795123] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 7437.797214] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7437.806696] Lustre: Skipped 1 previous similar message [ 7442.598115] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.3@tcp (stopping) [ 7442.602618] Lustre: Skipped 2 previous similar messages [ 7447.716603] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.3@tcp (stopping) [ 7447.721090] Lustre: Skipped 2 previous similar messages [ 7449.055123] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7449.107449] LustreError: 447719:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7449.135526] Lustre: server umount lustre-MDT0001 complete [ 7458.278417] LustreError: 444283:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7458.287553] LustreError: 444283:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7463.392860] LustreError: 444288:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7463.393683] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7463.401548] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7463.403066] Lustre: Skipped 2 previous similar messages [ 7468.516932] LustreError: 444289:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7473.634641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7473.636849] LustreError: 444289:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7473.638355] Lustre: Skipped 1 previous similar message [ 7476.191245] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7476.214183] LustreError: 447992:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 7476.217567] LustreError: 447992:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7476.268969] Lustre: server umount lustre-OST0000 complete [ 7484.304584] LustreError: 448194:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 7484.309793] LustreError: 448194:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7484.405170] Lustre: server umount lustre-MDT0000 complete [ 7490.658803] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 7491.995300] Key type lgssc unregistered [ 7492.137747] LNet: 448879:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7492.145441] LNetError: 448879:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7492.159724] LNet: Removed LNI 192.168.203.103@tcp [ 7492.565847] Key type .llcrypt unregistered [ 7492.567858] Key type ._llcrypt unregistered [ 7510.394528] Key type ._llcrypt registered [ 7510.396524] Key type .llcrypt registered [ 7510.459208] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 7510.616641] Key type .llcrypt unregistered [ 7510.618709] Key type ._llcrypt unregistered [ 7518.385527] Key type ._llcrypt registered [ 7518.386560] Key type .llcrypt registered [ 7518.424801] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 7518.748428] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7518.777281] alg: No test for adler32 (adler32-zlib) [ 7519.641823] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 7519.720138] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 7521.303182] Key type lgssc registered [ 7521.848409] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7525.304076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7528.164113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7531.048616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7534.202225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7534.227236] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7534.234185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7535.335885] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7535.350230] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7535.398170] Lustre: lustre-MDT0000: new disk, initializing [ 7535.434421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7535.446278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7536.996178] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 7542.544939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7542.581963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7542.638821] Lustre: 452199:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7542.680501] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7542.683926] Lustre: Skipped 1 previous similar message [ 7542.727739] Lustre: lustre-MDT0001: new disk, initializing [ 7542.775806] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7542.796870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7542.802784] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7544.400570] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 7548.372631] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7550.118057] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7553.401975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7553.444677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7553.574463] Lustre: lustre-OST0000: new disk, initializing [ 7553.579115] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7553.627529] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7556.118106] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 7558.154085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7558.159968] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7558.214878] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7559.996930] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7563.235186] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7563.236619] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7563.237217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7563.242385] Lustre: Skipped 2 previous similar messages [ 7568.352416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7568.354175] Lustre: Skipped 4 previous similar messages [ 7568.964185] LustreError: 454643:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7568.999772] Lustre: server umount lustre-MDT0000 complete [ 7570.267093] LustreError: 453946:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763088186 with bad export cookie 17867025685616875568 [ 7570.270777] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7570.272409] LustreError: 453946:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7570.339614] LustreError: 454844:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7570.342136] LustreError: 454844:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7570.386694] Lustre: server umount lustre-MDT0001 complete [ 7580.796299] LustreError: 455044:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7580.800111] LustreError: 455044:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7580.867467] Lustre: server umount lustre-OST0000 complete [ 7585.647280] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 21:43:22 (1763088202) [ 7586.139434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7589.429819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7589.660061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7591.456367] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 7594.976466] LustreError: 455872:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7594.982250] LustreError: 455872:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7595.339760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7595.465111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7597.133399] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 7599.525454] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7601.194719] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7604.388506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7604.435616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7604.538677] Lustre: Found index 0 for lustre-OST0000, updating log [ 7604.631738] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7607.115132] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 7609.641234] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7613.869452] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 7619.044643] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7619.046750] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7619.046978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7619.051069] Lustre: Skipped 1 previous similar message [ 7622.514151] LustreError: 458132:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7622.517892] LustreError: 458132:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7622.575887] Lustre: server umount lustre-OST0000 complete [ 7623.848680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7625.186253] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7625.187138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7625.193288] Lustre: Skipped 1 previous similar message [ 7625.196786] Lustre: Skipped 3 previous similar messages [ 7630.923971] LustreError: 458386:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7630.927629] LustreError: 458386:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7630.963286] Lustre: server umount lustre-MDT0000 complete [ 7631.328940] LustreError: 456586:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7631.337922] LustreError: 456586:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7632.747507] LustreError: 455853:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763088249 with bad export cookie 17867025685616876961 [ 7632.752875] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7632.758324] LustreError: 455853:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7632.898282] Lustre: server umount lustre-MDT0001 complete [ 7636.698464] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 21:44:13 (1763088253) [ 7637.175905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7637.658249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7638.135195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7638.627569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7639.126127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7639.590281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7640.079984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7640.562447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7641.044370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7641.535541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7642.026456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7642.499502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7642.967448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7643.409091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7643.930557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7644.418637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7644.884559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7645.313638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7645.784687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7646.200993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7646.687235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7647.163140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7647.675613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7648.175215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7648.667068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7649.161913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7649.597352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7650.059200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7650.506386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7650.926417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7651.383967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7651.850796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7652.292097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7652.777039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7653.266373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7653.747807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7654.198696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7654.661790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7655.061402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7655.527991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7655.991839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7656.464720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7656.962084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7657.449038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7657.876264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7658.273605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7658.641873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7659.120978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7659.582350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7660.073081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7660.580851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7664.091337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7664.323241] LustreError: 461973:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7664.360837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7666.053639] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 7669.728848] LustreError: 461973:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7670.046648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7671.944418] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 7674.436981] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7676.025248] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7679.370707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7679.419112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7679.525596] Lustre: Found index 0 for lustre-OST0000, updating log [ 7682.178747] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 7684.467294] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7687.138623] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7687.141713] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7687.152790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7687.156670] Lustre: Skipped 2 previous similar messages [ 7691.506201] LustreError: 464288:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7691.512629] LustreError: 464288:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 7691.572912] Lustre: server umount lustre-OST0000 complete [ 7694.177520] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7694.183814] Lustre: Skipped 1 previous similar message [ 7695.201269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7695.205020] Lustre: Skipped 4 previous similar messages [ 7699.296663] LustreError: 461991:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7699.303630] LustreError: 461991:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7699.310930] Lustre: server umount lustre-MDT0000 complete [ 7700.637173] LustreError: 463649:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763088316 with bad export cookie 17867025685616878032 [ 7700.643100] LustreError: 463649:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7700.643310] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7700.768114] Lustre: server umount lustre-MDT0001 complete [ 7701.720490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7708.751645] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 7712.504477] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 7716.579680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7719.525755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7722.503250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7725.613388] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7728.499742] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 21:45:45 (1763088345) [ 7742.873670] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 7744.097298] Key type lgssc unregistered [ 7744.294699] LNet: 468253:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7744.299618] LNetError: 468253:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7870.247437] LNet: Removed LNI 192.168.203.103@tcp [ 7870.559945] Key type .llcrypt unregistered [ 7870.561419] Key type ._llcrypt unregistered [ 7877.479912] Key type ._llcrypt registered [ 7877.481048] Key type .llcrypt registered [ 7877.529859] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 7877.933759] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7877.977095] alg: No test for adler32 (adler32-zlib) [ 7878.832676] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 7878.919782] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 7880.495165] Key type lgssc registered [ 7880.879432] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7882.346615] LNet: Added LNI 192.168.203.103@tcp100 [8/256/0/180] [ 7882.623579] LNet: Added LNI 192.168.203.103@tcp101 [8/256/0/180] [ 7882.916564] LNet: Added LNI 192.168.203.103@tcp102 [8/256/0/180] [ 7883.203494] LNet: Added LNI 192.168.203.103@tcp103 [8/256/0/180] [ 7883.498050] LNet: Added LNI 192.168.203.103@tcp104 [8/256/0/180] [ 7883.784777] LNet: Added LNI 192.168.203.103@tcp105 [8/256/0/180] [ 7884.167717] LNet: Added LNI 192.168.203.103@tcp106 [8/256/0/180] [ 7884.513864] LNet: Added LNI 192.168.203.103@tcp107 [8/256/0/180] [ 7884.796134] LNet: Added LNI 192.168.203.103@tcp108 [8/256/0/180] [ 7885.088264] LNet: Added LNI 192.168.203.103@tcp109 [8/256/0/180] [ 7885.375494] LNet: Added LNI 192.168.203.103@tcp110 [8/256/0/180] [ 7885.675158] LNet: Added LNI 192.168.203.103@tcp111 [8/256/0/180] [ 7885.992919] LNet: Added LNI 192.168.203.103@tcp112 [8/256/0/180] [ 7886.323051] LNet: Added LNI 192.168.203.103@tcp113 [8/256/0/180] [ 7886.603309] LNet: Added LNI 192.168.203.103@tcp114 [8/256/0/180] [ 7886.893291] LNet: Added LNI 192.168.203.103@tcp115 [8/256/0/180] [ 7887.166686] LNet: Added LNI 192.168.203.103@tcp116 [8/256/0/180] [ 7887.444260] LNet: Added LNI 192.168.203.103@tcp117 [8/256/0/180] [ 7887.714822] LNet: Added LNI 192.168.203.103@tcp118 [8/256/0/180] [ 7888.040527] LNet: Added LNI 192.168.203.103@tcp119 [8/256/0/180] [ 7888.351358] LNet: Added LNI 192.168.203.103@tcp120 [8/256/0/180] [ 7888.618809] LNet: Added LNI 192.168.203.103@tcp121 [8/256/0/180] [ 7888.976171] LNet: Added LNI 192.168.203.103@tcp122 [8/256/0/180] [ 7889.306327] LNet: Added LNI 192.168.203.103@tcp123 [8/256/0/180] [ 7889.637895] LNet: Added LNI 192.168.203.103@tcp124 [8/256/0/180] [ 7889.930774] LNet: Added LNI 192.168.203.103@tcp125 [8/256/0/180] [ 7890.269428] LNet: Added LNI 192.168.203.103@tcp126 [8/256/0/180] [ 7890.614229] LNet: Added LNI 192.168.203.103@tcp127 [8/256/0/180] [ 7890.932914] LNet: Added LNI 192.168.203.103@tcp128 [8/256/0/180] [ 7891.204680] LNet: Added LNI 192.168.203.103@tcp129 [8/256/0/180] [ 7891.493995] LNet: Added LNI 192.168.203.103@tcp130 [8/256/0/180] [ 7891.802672] LNet: Added LNI 192.168.203.103@tcp131 [8/256/0/180] [ 7892.126925] LNet: Added LNI 192.168.203.103@tcp132 [8/256/0/180] [ 7892.402029] LNet: Added LNI 192.168.203.103@tcp133 [8/256/0/180] [ 7892.682535] LNet: Added LNI 192.168.203.103@tcp134 [8/256/0/180] [ 7892.964478] LNet: Added LNI 192.168.203.103@tcp135 [8/256/0/180] [ 7895.938101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7895.964967] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7895.973314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7897.036637] Lustre: 471241:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.103@tcp: can't add NID 192.168.203.103@tcp130: rc = -75 [ 7897.087645] LNetError: 471269:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.103 [ 7897.090533] LNetError: Protocol error connecting to 192.168.203.103@tcp115 at host 192.168.203.103:988: is it running a compatible version of Lustre? [ 7897.093386] LNetError: Skipped 1 previous similar message [ 7897.165386] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7897.182697] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7897.231352] Lustre: lustre-MDT0000: new disk, initializing [ 7897.264106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7897.270611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7898.153233] LNetError: 471269:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.103 [ 7898.158695] LNetError: 471269:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 2 previous similar messages [ 7898.162951] LNetError: Protocol error connecting to 192.168.203.103@tcp102 at host 192.168.203.103:988: is it running a compatible version of Lustre? [ 7899.112533] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 7904.544653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7904.587368] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7904.659713] Lustre: 471259:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7904.749832] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7904.750398] Lustre: 472048:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.103@tcp: can't add NID 192.168.203.103@tcp130: rc = -75 [ 7904.752650] Lustre: Skipped 1 previous similar message [ 7904.757254] Lustre: 472048:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 7904.772458] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7904.813835] Lustre: lustre-MDT0001: new disk, initializing [ 7904.900312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7904.924946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7904.930977] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7906.614520] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 7910.553447] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7912.195753] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7914.973215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7915.008332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7915.118531] Lustre: lustre-OST0000: new disk, initializing [ 7915.121275] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7915.164768] Lustre: 473020:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.103@tcp: can't add NID 192.168.203.103@tcp130: rc = -75 [ 7915.167812] Lustre: 473020:0:(lustre_peer.c:105:class_add_uuid()) Skipped 17 previous similar messages [ 7915.184899] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7917.218876] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 7920.861561] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7924.055191] LustreError: 473845:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7924.115550] Lustre: server umount lustre-OST0000 complete [ 7924.222130] Lustre: 473859:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.103@tcp: can't add NID 192.168.203.103@tcp130: rc = -75 [ 7924.227729] Lustre: 473859:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 7927.073455] LNetError: 471263:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.103 [ 7927.082157] LNetError: Protocol error connecting to 192.168.203.103@tcp118 at host 192.168.203.103:988: is it running a compatible version of Lustre? [ 7929.314450] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7929.316388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7929.321811] Lustre: Skipped 1 previous similar message [ 7929.328967] Lustre: Skipped 1 previous similar message [ 7932.052420] LustreError: 474053:0:(obd_class.h:479:obd_check_dev()) Device 17 not setup [ 7932.056301] LustreError: 474053:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7932.103881] Lustre: server umount lustre-MDT0000 complete [ 7932.193411] LNetError: 471262:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.103 [ 7932.197151] LNetError: Protocol error connecting to 192.168.203.103@tcp119 at host 192.168.203.103:988: is it running a compatible version of Lustre? [ 7934.000667] LustreError: 472046:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763088550 with bad export cookie 12110523317170241199 [ 7934.008783] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7934.113903] LustreError: 474255:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7934.115995] LustreError: 474255:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7934.183729] Lustre: server umount lustre-MDT0001 complete [ 7938.336752] LNetError: 471266:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.103 [ 7938.342638] LNetError: Protocol error connecting to 192.168.203.103@tcp115 at host 192.168.203.103:988: is it running a compatible version of Lustre? [ 7939.366783] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 7941.041608] Key type lgssc unregistered [ 7941.257593] LNet: 474794:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7941.263222] LNetError: 474794:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7941.268382] LNet: Removed LNI 192.168.203.103@tcp [ 7942.303196] LNet: Removed LNI 192.168.203.103@tcp100 [ 7942.305879] LNetError: 474794:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7942.310501] LNetError: 474794:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7942.315675] LNet: Removed LNI 192.168.203.103@tcp101 [ 7943.327284] LNet: Removed LNI 192.168.203.103@tcp102 [ 7943.330097] LNetError: 474794:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7943.334798] LNetError: 474794:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7943.339670] LNet: Removed LNI 192.168.203.103@tcp103 [ 7944.351250] LNet: Removed LNI 192.168.203.103@tcp104 [ 7945.375278] LNet: Removed LNI 192.168.203.103@tcp105 [ 7945.377595] LNetError: 474794:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7945.381279] LNetError: 474794:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 7945.385165] LNet: Removed LNI 192.168.203.103@tcp106 [ 7946.399258] LNet: Removed LNI 192.168.203.103@tcp107 [ 7947.423231] LNet: Removed LNI 192.168.203.103@tcp108 [ 7948.447230] LNet: Removed LNI 192.168.203.103@tcp109 [ 7949.471263] LNet: Removed LNI 192.168.203.103@tcp110 [ 7949.474092] LNetError: 474794:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7949.478734] LNetError: 474794:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 7949.483761] LNet: Removed LNI 192.168.203.103@tcp111 [ 7950.495201] LNet: Removed LNI 192.168.203.103@tcp112 [ 7951.519208] LNet: Removed LNI 192.168.203.103@tcp113 [ 7952.543253] LNet: Removed LNI 192.168.203.103@tcp114 [ 7953.567296] LNet: Removed LNI 192.168.203.103@tcp115 [ 7954.591277] LNet: Removed LNI 192.168.203.103@tcp116 [ 7955.615232] LNet: Removed LNI 192.168.203.103@tcp117 [ 7956.639196] LNet: Removed LNI 192.168.203.103@tcp118 [ 7957.663212] LNet: Removed LNI 192.168.203.103@tcp119 [ 7957.666137] LNetError: 474794:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7957.671016] LNetError: 474794:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 7957.676384] LNet: Removed LNI 192.168.203.103@tcp120 [ 7957.679212] LNet: Removed LNI 192.168.203.103@tcp121 [ 7957.682443] LNet: Removed LNI 192.168.203.103@tcp122 [ 7957.685737] LNet: Removed LNI 192.168.203.103@tcp123 [ 7957.688275] LNet: Removed LNI 192.168.203.103@tcp124 [ 7957.690916] LNet: Removed LNI 192.168.203.103@tcp125 [ 7957.693244] LNet: Removed LNI 192.168.203.103@tcp126 [ 7957.695361] LNet: Removed LNI 192.168.203.103@tcp127 [ 7957.697628] LNet: Removed LNI 192.168.203.103@tcp128 [ 7957.700422] LNet: Removed LNI 192.168.203.103@tcp129 [ 7957.702899] LNet: Removed LNI 192.168.203.103@tcp130 [ 7957.705169] LNet: Removed LNI 192.168.203.103@tcp131 [ 7957.707454] LNet: Removed LNI 192.168.203.103@tcp132 [ 7957.709636] LNet: Removed LNI 192.168.203.103@tcp133 [ 7957.711763] LNet: Removed LNI 192.168.203.103@tcp134 [ 7957.719953] LNet: Removed LNI 192.168.203.103@tcp135 [ 7958.051037] Key type .llcrypt unregistered [ 7958.052171] Key type ._llcrypt unregistered [ 7968.357838] Key type ._llcrypt registered [ 7968.360016] Key type .llcrypt registered [ 7968.426830] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 7976.523637] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 7976.854342] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7976.892115] alg: No test for adler32 (adler32-zlib) [ 7977.735650] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 7977.808114] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 7979.383193] Key type lgssc registered [ 7979.950755] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7984.401308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7987.739459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7990.854296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7993.832822] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7996.987346] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 21:50:13 (1763088613) [ 8004.990311] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 8006.369596] Key type lgssc unregistered [ 8006.545682] LNet: 478811:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8006.550486] LNetError: 478811:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8006.564621] LNet: Removed LNI 192.168.203.103@tcp [ 8007.049639] Key type .llcrypt unregistered [ 8007.051878] Key type ._llcrypt unregistered [ 8015.358850] Key type ._llcrypt registered [ 8015.360885] Key type .llcrypt registered [ 8015.421935] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 8015.873310] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8015.899041] alg: No test for adler32 (adler32-zlib) [ 8016.802298] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 8016.909554] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 8018.519187] Key type lgssc registered [ 8019.171301] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8023.958715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8023.992993] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8024.005506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8025.134186] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8025.148123] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8025.191176] Lustre: lustre-MDT0000: new disk, initializing [ 8025.223168] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8025.233071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8026.918034] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 8032.593103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8032.638306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8032.699898] Lustre: 479982:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8032.747345] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8032.750857] Lustre: Skipped 1 previous similar message [ 8032.800094] Lustre: lustre-MDT0001: new disk, initializing [ 8032.846119] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8032.867580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8032.872501] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8034.834750] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 8038.916613] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8040.649961] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8043.984982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8044.022226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8044.195446] Lustre: lustre-OST0000: new disk, initializing [ 8044.198775] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8044.259740] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8044.570618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8044.576794] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8044.637595] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8047.010697] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 8051.224636] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8052.676866] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing lnet_if_list [ 8054.838504] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 8054.854642] LNet: Added LNI 192.168.203.103@tcp10 [8/256/0/180] [ 8057.504707] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 8057.523602] LNet: Added LNI 192.168.203.103@tcp11 [8/256/0/180] [ 8059.238367] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 8059.256826] LNet: Added LNI 192.168.203.103@tcp12 [8/256/0/180] [ 8061.064886] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 8061.078206] LNet: Added LNI 192.168.203.103@tcp13 [8/256/0/180] [ 8062.824331] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 8062.837763] LNet: Added LNI 192.168.203.103@tcp14 [8/256/0/180] [ 8064.752303] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 8064.765818] LNet: Added LNI 192.168.203.103@tcp15 [8/256/0/180] [ 8066.832750] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 8066.851899] LNet: Added LNI 192.168.203.103@tcp16 [8/256/0/180] [ 8069.140955] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 8069.152949] LNet: Added LNI 192.168.203.103@tcp17 [8/256/0/180] [ 8070.960571] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 8070.979130] LNet: Added LNI 192.168.203.103@tcp18 [8/256/0/180] [ 8073.124474] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 8073.138795] LNet: Added LNI 192.168.203.103@tcp19 [8/256/0/180] [ 8075.393182] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 8075.410817] LNet: Added LNI 192.168.203.103@tcp20 [8/256/0/180] [ 8077.723924] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 8077.743561] LNet: Added LNI 192.168.203.103@tcp21 [8/256/0/180] [ 8080.180961] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 8080.203334] LNet: Added LNI 192.168.203.103@tcp22 [8/256/0/180] [ 8082.622220] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 8082.640956] LNet: Added LNI 192.168.203.103@tcp23 [8/256/0/180] [ 8085.006734] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 8085.022568] LNet: Added LNI 192.168.203.103@tcp24 [8/256/0/180] [ 8087.239671] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 8087.250822] LNet: Added LNI 192.168.203.103@tcp25 [8/256/0/180] [ 8089.405867] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 8089.421377] LNet: Added LNI 192.168.203.103@tcp26 [8/256/0/180] [ 8091.917124] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 8091.935677] LNet: Added LNI 192.168.203.103@tcp27 [8/256/0/180] [ 8091.950092] LNetError: 479244:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.103 [ 8091.958416] LNetError: Protocol error connecting to 192.168.203.103@tcp26 at host 192.168.203.103:988: is it running a compatible version of Lustre? [ 8094.165924] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 8094.180542] LNet: Added LNI 192.168.203.103@tcp28 [8/256/0/180] [ 8094.184243] LNetError: 479243:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.103 [ 8094.188634] LNetError: Protocol error connecting to 192.168.203.103@tcp26 at host 192.168.203.103:988: is it running a compatible version of Lustre? [ 8096.447881] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 8096.463610] LNet: Added LNI 192.168.203.103@tcp29 [8/256/0/180] [ 8096.470107] LNetError: 479242:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.103 [ 8096.475502] LNetError: Protocol error connecting to 192.168.203.103@tcp28 at host 192.168.203.103:988: is it running a compatible version of Lustre? [ 8096.799112] LNet: 2 local NIs in recovery (showing 2): 192.168.203.103@tcp28, 192.168.203.103@tcp29 [ 8098.555816] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 8098.570579] LNet: Added LNI 192.168.203.103@tcp30 [8/256/0/180] [ 8099.874164] LNetError: 479244:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.103 [ 8099.880525] LNetError: 479244:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 3 previous similar messages [ 8099.885137] LNetError: Protocol error connecting to 192.168.203.103@tcp28 at host 192.168.203.103:988: is it running a compatible version of Lustre? [ 8099.892819] LNetError: Skipped 3 previous similar messages [ 8100.776282] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 8100.784948] LNet: Added LNI 192.168.203.103@tcp31 [8/256/0/180] [ 8103.408892] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 8103.432318] LNet: Added LNI 192.168.203.103@tcp32 [8/256/0/180] [ 8103.975845] LNetError: 479244:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.103 [ 8103.981430] LNetError: 479244:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 3 previous similar messages [ 8103.985787] LNetError: Protocol error connecting to 192.168.203.103@tcp30 at host 192.168.203.103:988: is it running a compatible version of Lustre? [ 8103.990538] LNetError: Skipped 3 previous similar messages [ 8106.092078] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 8106.106831] LNet: Added LNI 192.168.203.103@tcp33 [8/256/0/180] [ 8108.459338] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 8108.475519] LNet: Added LNI 192.168.203.103@tcp34 [8/256/0/180] [ 8111.284010] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 8111.301167] LNet: Added LNI 192.168.203.103@tcp35 [8/256/0/180] [ 8112.165547] LNetError: 479242:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.103 [ 8112.170131] LNetError: 479242:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 8 previous similar messages [ 8112.172768] LNetError: Protocol error connecting to 192.168.203.103@tcp33 at host 192.168.203.103:988: is it running a compatible version of Lustre? [ 8112.176226] LNetError: Skipped 8 previous similar messages [ 8114.131349] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 8114.150481] LNet: Added LNI 192.168.203.103@tcp36 [8/256/0/180] [ 8117.481887] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 8117.502161] LNet: Added LNI 192.168.203.103@tcp37 [8/256/0/180] [ 8120.886710] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 8120.899893] LNet: Added LNI 192.168.203.103@tcp38 [8/256/0/180] [ 8124.283260] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 8124.302826] LNet: Added LNI 192.168.203.103@tcp39 [8/256/0/180] [ 8127.683937] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 8127.708495] LNet: Added LNI 192.168.203.103@tcp40 [8/256/0/180] [ 8128.545916] LNetError: 496546:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.103 [ 8128.549706] LNetError: 496546:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 30 previous similar messages [ 8128.552686] LNetError: Protocol error connecting to 192.168.203.103@tcp29 at host 192.168.203.103:988: is it running a compatible version of Lustre? [ 8128.556914] LNetError: Skipped 30 previous similar messages [ 8131.043673] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 8131.055962] LNet: Added LNI 192.168.203.103@tcp41 [8/256/0/180] [ 8134.320932] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 8134.337571] LNet: Added LNI 192.168.203.103@tcp42 [8/256/0/180] [ 8138.198820] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 8138.225874] LNet: Added LNI 192.168.203.103@tcp43 [8/256/0/180] [ 8142.119746] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 8142.138696] LNet: Added LNI 192.168.203.103@tcp44 [8/256/0/180] [ 8146.232518] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 8146.252325] LNet: Added LNI 192.168.203.103@tcp45 [8/256/0/180] [ 8149.920488] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 8149.941742] LNet: Added LNI 192.168.203.103@tcp46 [8/256/0/180] [ 8153.497200] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 8153.508760] LNet: Added LNI 192.168.203.103@tcp47 [8/256/0/180] [ 8156.460299] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 8156.473225] LNet: Added LNI 192.168.203.103@tcp48 [8/256/0/180] [ 8157.216264] LNet: 8 local NIs in recovery (showing 8): 192.168.203.103@tcp44, 192.168.203.103@tcp38, 192.168.203.103@tcp43, 192.168.203.103@tcp36, 192.168.203.103@tcp40, 192.168.203.103@tcp34, 192.168.203.103@tcp33, 192.168.203.103@tcp45 [ 8159.845604] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 8159.861618] LNet: Added LNI 192.168.203.103@tcp49 [8/256/0/180] [ 8161.313926] LNetError: 479244:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.103 [ 8161.318914] LNetError: 479244:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 36 previous similar messages [ 8161.323079] LNetError: Protocol error connecting to 192.168.203.103@tcp43 at host 192.168.203.103:988: is it running a compatible version of Lustre? [ 8161.328211] LNetError: Skipped 36 previous similar messages [ 8163.513362] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 8163.531108] LNet: Added LNI 192.168.203.103@tcp50 [8/256/0/180] [ 8177.635122] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8177.637799] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8177.641518] LustreError: Skipped 1 previous similar message [ 8177.642890] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8177.648465] Lustre: Skipped 1 previous similar message [ 8177.656746] Lustre: Skipped 1 previous similar message [ 8179.055160] LustreError: 504180:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8179.111483] Lustre: server umount lustre-OST0000 complete [ 8181.216548] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8181.220617] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8181.227439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8182.753296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8182.754560] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8187.087040] LustreError: 504383:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8187.090724] LustreError: 504383:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8187.127418] Lustre: server umount lustre-MDT0000 complete [ 8188.872865] LustreError: 479972:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763088805 with bad export cookie 13142081991184903332 [ 8188.878261] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8188.879479] LustreError: 479972:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8188.987049] LustreError: 504587:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8188.990456] LustreError: 504587:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8189.043336] Lustre: server umount lustre-MDT0001 complete [ 8193.538277] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 8195.017187] Key type lgssc unregistered [ 8195.174511] LNet: 505126:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8195.176809] LNetError: 505126:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8195.178723] LNet: Removed LNI 192.168.203.103@tcp [ 8196.191179] LNet: Removed LNI 192.168.203.103@tcp10 [ 8196.192354] LNetError: 505126:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8196.194105] LNetError: 505126:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 8196.195942] LNet: Removed LNI 192.168.203.103@tcp11 [ 8197.215235] LNet: Removed LNI 192.168.203.103@tcp12 [ 8197.218176] LNetError: 505126:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8197.222828] LNetError: 505126:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 8197.227983] LNet: Removed LNI 192.168.203.103@tcp13 [ 8198.239297] LNet: Removed LNI 192.168.203.103@tcp14 [ 8199.263136] LNet: Removed LNI 192.168.203.103@tcp15 [ 8199.264273] LNetError: 505126:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8199.265899] LNetError: 505126:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 8199.267754] LNet: Removed LNI 192.168.203.103@tcp16 [ 8200.287178] LNet: Removed LNI 192.168.203.103@tcp17 [ 8201.311243] LNet: Removed LNI 192.168.203.103@tcp18 [ 8202.335216] LNet: Removed LNI 192.168.203.103@tcp19 [ 8203.359223] LNet: Removed LNI 192.168.203.103@tcp20 [ 8203.361696] LNetError: 505126:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8203.365902] LNetError: 505126:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 8203.370492] LNet: Removed LNI 192.168.203.103@tcp21 [ 8204.383228] LNet: Removed LNI 192.168.203.103@tcp22 [ 8205.407248] LNet: Removed LNI 192.168.203.103@tcp23 [ 8206.431232] LNet: Removed LNI 192.168.203.103@tcp24 [ 8207.455199] LNet: Removed LNI 192.168.203.103@tcp25 [ 8208.479169] LNet: Removed LNI 192.168.203.103@tcp26 [ 8209.503224] LNet: Removed LNI 192.168.203.103@tcp27 [ 8210.527216] LNet: Removed LNI 192.168.203.103@tcp28 [ 8211.551227] LNet: Removed LNI 192.168.203.103@tcp29 [ 8211.553817] LNetError: 505126:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8211.558028] LNetError: 505126:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 8211.562542] LNet: Removed LNI 192.168.203.103@tcp30 [ 8212.575223] LNet: Removed LNI 192.168.203.103@tcp31 [ 8213.599281] LNet: Removed LNI 192.168.203.103@tcp32 [ 8214.623207] LNet: Removed LNI 192.168.203.103@tcp33 [ 8214.625905] LNet: Removed LNI 192.168.203.103@tcp34 [ 8215.647193] LNet: Removed LNI 192.168.203.103@tcp35 [ 8216.671212] LNet: Removed LNI 192.168.203.103@tcp36 [ 8217.695142] LNet: Removed LNI 192.168.203.103@tcp37 [ 8217.698287] LNet: Removed LNI 192.168.203.103@tcp38 [ 8218.719196] LNet: Removed LNI 192.168.203.103@tcp39 [ 8218.723967] LNet: Removed LNI 192.168.203.103@tcp40 [ 8219.743142] LNet: Removed LNI 192.168.203.103@tcp41 [ 8220.767154] LNet: Removed LNI 192.168.203.103@tcp42 [ 8221.791296] LNet: Removed LNI 192.168.203.103@tcp43 [ 8221.796081] LNet: Removed LNI 192.168.203.103@tcp44 [ 8221.800388] LNet: Removed LNI 192.168.203.103@tcp45 [ 8222.815179] LNet: Removed LNI 192.168.203.103@tcp46 [ 8223.839239] LNet: Removed LNI 192.168.203.103@tcp47 [ 8224.863199] LNet: Removed LNI 192.168.203.103@tcp48 [ 8225.887185] LNet: Removed LNI 192.168.203.103@tcp49 [ 8225.897837] LNet: Removed LNI 192.168.203.103@tcp50 [ 8226.285406] LNetError: 505188:0:(module.c:692:libcfs_exit()) Portals memory leaked: 376 bytes [ 8226.309791] Key type .llcrypt unregistered [ 8226.310572] Key type ._llcrypt unregistered [ 8234.967345] Key type ._llcrypt registered [ 8234.968975] Key type .llcrypt registered [ 8235.028273] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 8235.320404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8235.327947] alg: No test for adler32 (adler32-zlib) [ 8236.207752] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 8236.310028] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 8237.903171] Key type lgssc registered [ 8238.379383] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8242.037963] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8242.047308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8243.227661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8244.817093] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 8248.906952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8249.068198] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8250.711916] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 8253.178738] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8254.837920] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8258.057892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8258.256784] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8260.728184] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 8263.223149] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8278.157460] loop0: detected capacity change from 0 to 204800000 [ 8278.274036] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8278.279686] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8278.284978] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8278.305118] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8281.211356] loop0: detected capacity change from 0 to 204800000 [ 8281.330782] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8281.333550] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8281.337625] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8281.411777] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8283.322609] loop0: detected capacity change from 0 to 204800000 [ 8285.897169] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8285.934906] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8286.002594] Lustre: fs15246-OST0000: new disk, initializing [ 8286.006351] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 8286.039488] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8288.450024] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 8293.472355] loop1: detected capacity change from 0 to 204800000 [ 8296.151530] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8296.190715] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8296.242745] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 8296.271697] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 8296.313866] Lustre: fs15246-MDT0000: new disk, initializing [ 8296.383473] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8296.396142] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8296.446341] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 8297.503830] Lustre: 510742:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@000000004c188335 x1848732656109440/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1763088929 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 8298.243836] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 8306.594234] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 8306.599695] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 8306.612201] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 8307.422690] LustreError: 511869:0:(obd_class.h:479:obd_check_dev()) Device 30 not setup [ 8307.452854] Lustre: server umount fs15246-MDT0000 complete [ 8311.272716] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8311.280274] LustreError: 507066:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8311.287566] LustreError: 507066:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8311.858139] LustreError: 512370:0:(obd_class.h:479:obd_check_dev()) Device 32 not setup [ 8311.862356] LustreError: 512370:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8311.887917] Lustre: server umount fs15246-OST0000 complete [ 8319.969143] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8319.973580] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8319.981343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8320.993307] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8320.997812] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8321.005797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8322.675220] LustreError: 512871:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8322.680031] LustreError: 512871:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 8322.736537] Lustre: server umount lustre-OST0000 complete [ 8326.112496] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8326.119199] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8326.127737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8326.131424] Lustre: Skipped 2 previous similar messages [ 8330.702745] LustreError: 513073:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8330.706471] LustreError: 513073:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8330.741899] Lustre: server umount lustre-MDT0000 complete [ 8331.743941] LustreError: 507494:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8331.747888] LustreError: 507494:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8332.019073] LustreError: 506328:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763088948 with bad export cookie 1491120955226950927 [ 8332.021867] LustreError: 506328:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8332.022505] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8332.124628] Lustre: server umount lustre-MDT0001 complete [ 8336.242273] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 8337.090277] Key type lgssc unregistered [ 8337.207344] LNet: 513805:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8337.209222] LNetError: 513805:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8337.218304] LNet: Removed LNI 192.168.203.103@tcp [ 8337.460482] Key type .llcrypt unregistered [ 8337.461404] Key type ._llcrypt unregistered [ 8346.466699] Key type ._llcrypt registered [ 8346.468285] Key type .llcrypt registered [ 8346.525607] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 8346.960786] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8347.011932] alg: No test for adler32 (adler32-zlib) [ 8347.858623] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 8347.941063] LNet: Added LNI 192.168.203.103@tcp [8/256/0/180] [ 8349.543155] Key type lgssc registered [ 8349.985974] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8353.269481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8355.344912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8357.744528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8359.354636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8365.318435] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 8369.391983] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 8373.896401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8376.974813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8379.948417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8383.038321] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8385.786767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8385.815906] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8385.828697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8386.934246] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8386.943861] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8386.978962] Lustre: lustre-MDT0000: new disk, initializing [ 8387.003381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8387.009513] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8388.505856] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 8393.914995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8393.942198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8394.003560] Lustre: 518215:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8394.042406] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8394.045418] Lustre: Skipped 1 previous similar message [ 8394.093408] Lustre: lustre-MDT0001: new disk, initializing [ 8394.131860] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8394.147331] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8394.152894] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8395.627356] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 8399.539376] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8401.253527] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8404.479752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8404.518421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8404.681797] Lustre: lustre-OST0000: new disk, initializing [ 8404.684519] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8404.740083] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8405.699735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8405.703848] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8405.750766] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8407.191522] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_default_debug -1 all [ 8411.188906] Lustre: DEBUG MARKER: oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8413.056371] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8413.188006] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8415.020686] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8415.131911] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8416.227213] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8416.229118] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8416.231377] LustreError: Skipped 1 previous similar message [ 8416.232323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8416.237658] Lustre: Skipped 1 previous similar message [ 8416.246192] Lustre: Skipped 1 previous similar message [ 8421.348837] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8421.353394] Lustre: Skipped 1 previous similar message [ 8422.067127] LustreError: 521155:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8422.131384] Lustre: server umount lustre-OST0000 complete [ 8424.928501] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8424.932709] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8424.940136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8426.466469] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8431.585144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8431.588745] Lustre: Skipped 3 previous similar messages [ 8436.704570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8436.707252] Lustre: Skipped 1 previous similar message [ 8438.239143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8438.295911] LustreError: 521356:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8438.299558] LustreError: 521356:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8438.337535] Lustre: server umount lustre-MDT0000 complete [ 8439.945475] LustreError: 518997:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763089056 with bad export cookie 1141916197841415668 [ 8439.948865] LustreError: MGC192.168.203.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8439.949757] LustreError: 518997:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8440.045485] LustreError: 521559:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8440.048678] LustreError: 521559:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8440.111817] Lustre: server umount lustre-MDT0001 complete [ 8450.646991] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing set_hostid [ 8454.740817] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing load_modules_local [ 8458.896834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8461.991461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8464.757907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8467.544182] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8468.196597] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8263 sec ========== 21:58:05 (1763089085) [ 8468.977430] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 21:58:05 (1763089085) === [ 8470.073743] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 21:58:07 (1763089087) === [ 8487.436205] Lustre: DEBUG MARKER: oleg303-server.virtnet: executing unload_modules_local [ 8488.966590] Key type lgssc unregistered [ 8489.132694] LNet: 526407:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8489.137430] LNetError: 526407:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8489.149598] LNet: Removed LNI 192.168.203.103@tcp [ 8489.584956] Key type .llcrypt unregistered [ 8489.586598] Key type ._llcrypt unregistered