[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 482712996 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002423] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004017] kvm-guest: setup PV IPIs [ 0.007644] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.010135] LSM: Security Framework initializing [ 0.011055] Yama: becoming mindful. [ 0.012048] SELinux: Initializing. [ 0.013090] *** VALIDATE selinux *** [ 0.021887] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026578] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028120] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030097] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031131] *** VALIDATE tmpfs *** [ 0.032435] *** VALIDATE proc *** [ 0.034179] *** VALIDATE cgroup *** [ 0.035012] *** VALIDATE cgroup2 *** [ 0.036280] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037174] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039000] Spectre V2 : User space: Vulnerable [ 0.039011] Speculative Store Bypass: Vulnerable [ 0.042489] debug: unmapping init [mem 0xffffffffa3259000-0xffffffffa3260fff] [ 0.044160] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045683] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046024] ... version: 2 [ 0.047013] ... bit width: 48 [ 0.048010] ... generic registers: 4 [ 0.048990] ... value mask: 0000ffffffffffff [ 0.049014] ... max period: 00007fffffffffff [ 0.050010] ... fixed-purpose events: 3 [ 0.051011] ... event mask: 000000070000000f [ 0.052279] rcu: Hierarchical SRCU implementation. [ 0.054280] smp: Bringing up secondary CPUs ... [ 0.055609] x86: Booting SMP configuration: [ 0.056021] .... node #0, CPUs: #1 #2 #3 [ 0.059143] smp: Brought up 1 node, 4 CPUs [ 0.061012] smpboot: Max logical packages: 1 [ 0.062017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.183751] node 0 deferred pages initialised in 119ms [ 0.186520] devtmpfs: initialized [ 0.187361] x86/mm: Memory block size: 128MB [ 0.189933] gcov: version magic: 0x41383552 [ 0.192230] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.195078] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.197305] pinctrl core: initialized pinctrl subsystem [ 0.198139] [ 0.198555] ************************************************************* [ 0.200010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.202015] ** ** [ 0.205016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.207012] ** ** [ 0.209013] ** This means that this kernel is built to expose internal ** [ 0.210010] ** IOMMU data structures, which may compromise security on ** [ 0.212011] ** your system. ** [ 0.213007] ** ** [ 0.215014] ** If you see this message and you are not debugging the ** [ 0.217012] ** kernel, report this immediately to your vendor! ** [ 0.219013] ** ** [ 0.221014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.223019] ************************************************************* [ 0.225670] NET: Registered protocol family 16 [ 0.227463] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.229061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.232108] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.236031] cpuidle: using governor menu [ 0.237519] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.240454] PCI: Using configuration type 1 for base access [ 0.242123] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.250296] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.252046] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.257068] cryptd: max_cpu_qlen set to 1000 [ 0.260215] ACPI: Added _OSI(Module Device) [ 0.261014] ACPI: Added _OSI(Processor Device) [ 0.263017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.265033] ACPI: Added _OSI(Processor Aggregator Device) [ 0.269000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.274377] ACPI: Interpreter enabled [ 0.276059] ACPI: PM: (supports S0 S3 S4 S5) [ 0.277029] ACPI: Using IOAPIC for interrupt routing [ 0.279169] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.283385] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.291682] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.293027] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.295014] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.298068] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.301076] acpiphp: Slot [2] registered [ 0.302089] acpiphp: Slot [3] registered [ 0.303038] acpiphp: Slot [4] registered [ 0.303919] acpiphp: Slot [5] registered [ 0.305078] acpiphp: Slot [6] registered [ 0.306008] acpiphp: Slot [7] registered [ 0.306981] acpiphp: Slot [8] registered [ 0.308076] acpiphp: Slot [9] registered [ 0.309045] acpiphp: Slot [10] registered [ 0.310091] acpiphp: Slot [11] registered [ 0.311085] acpiphp: Slot [12] registered [ 0.312088] acpiphp: Slot [13] registered [ 0.314127] acpiphp: Slot [14] registered [ 0.315095] acpiphp: Slot [15] registered [ 0.316064] acpiphp: Slot [16] registered [ 0.318096] acpiphp: Slot [17] registered [ 0.319086] acpiphp: Slot [18] registered [ 0.321083] acpiphp: Slot [19] registered [ 0.322119] acpiphp: Slot [20] registered [ 0.323084] acpiphp: Slot [21] registered [ 0.325083] acpiphp: Slot [22] registered [ 0.326087] acpiphp: Slot [23] registered [ 0.328083] acpiphp: Slot [24] registered [ 0.329000] acpiphp: Slot [25] registered [ 0.329000] acpiphp: Slot [26] registered [ 0.331128] acpiphp: Slot [27] registered [ 0.334124] acpiphp: Slot [28] registered [ 0.335095] acpiphp: Slot [29] registered [ 0.336100] acpiphp: Slot [30] registered [ 0.338061] acpiphp: Slot [31] registered [ 0.339051] PCI host bridge to bus 0000:00 [ 0.340021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.343028] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.345022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.348024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.350021] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.352034] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.354173] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.357108] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.361501] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.370015] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.375078] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.379021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.381017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.383021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.385501] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.388827] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.390048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.393748] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.399016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.416041] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.425025] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.433339] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.439025] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.448028] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.462037] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.471447] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.477020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.482020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.504027] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.513685] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.520028] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.526019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.541053] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.551286] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.557026] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.562022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.574026] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.583498] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.591018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.597026] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.608018] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.616677] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.629023] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.634019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.648020] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.663073] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.665354] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.667444] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.670451] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.672288] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.676331] iommu: Default domain type: Passthrough [ 0.678442] SCSI subsystem initialized [ 0.680189] ACPI: bus type USB registered [ 0.681119] usbcore: registered new interface driver usbfs [ 0.683080] usbcore: registered new interface driver hub [ 0.685125] usbcore: registered new device driver usb [ 0.687174] pps_core: LinuxPPS API ver. 1 registered [ 0.689015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.692082] PTP clock support registered [ 0.695060] EDAC MC: Ver: 3.0.0 [ 0.697120] PCI: Using ACPI for IRQ routing [ 0.698883] NetLabel: Initializing [ 0.700021] NetLabel: domain hash size = 128 [ 0.702012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.704082] NetLabel: unlabeled traffic allowed by default [ 0.706141] vgaarb: loaded [ 0.707297] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.709018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.714438] clocksource: Switched to clocksource kvm-clock [ 0.820064] VFS: Disk quotas dquot_6.6.0 [ 0.822743] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.827486] *** VALIDATE ramfs *** [ 0.829725] *** VALIDATE hugetlbfs *** [ 0.832478] pnp: PnP ACPI init [ 0.834936] pnp: PnP ACPI: found 6 devices [ 0.856715] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.860077] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.861964] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.864021] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.866282] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.868303] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.870562] NET: Registered protocol family 2 [ 0.872898] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.877528] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.881023] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.886244] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.889652] TCP: Hash tables configured (established 65536 bind 65536) [ 0.892442] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.895569] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.898436] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.901403] NET: Registered protocol family 1 [ 0.904375] RPC: Registered named UNIX socket transport module. [ 0.907023] RPC: Registered udp transport module. [ 0.908857] RPC: Registered tcp transport module. [ 0.910669] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.913398] NET: Registered protocol family 44 [ 0.915246] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.917464] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.919136] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.920929] PCI: CLS 0 bytes, default 64 [ 0.922212] Unpacking initramfs... [ 2.326984] debug: unmapping init [mem 0xffff9cf93cc54000-0xffff9cf93ffbffff] [ 2.331738] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.334974] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.338797] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.856521] Initialise system trusted keyrings [ 2.858298] Key type blacklist registered [ 2.860356] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.869506] zbud: loaded [ 2.872516] *** VALIDATE nfs *** [ 2.877414] *** VALIDATE nfs4 *** [ 2.879130] pstore: using deflate compression [ 2.882102] Platform Keyring initialized [ 2.996899] NET: Registered protocol family 38 [ 2.998852] Key type asymmetric registered [ 3.000606] Asymmetric key parser 'x509' registered [ 3.002390] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.005475] io scheduler mq-deadline registered [ 3.007020] io scheduler kyber registered [ 3.008693] io scheduler bfq registered [ 3.010717] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.013246] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.015186] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.024050] ACPI: Power Button [PWRF] [ 3.115734] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.218565] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.402695] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.494332] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.676372] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.708030] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.743888] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.749485] Non-volatile memory driver v1.3 [ 3.751186] Linux agpgart interface v0.103 [ 3.783045] virtio_blk virtio1: [vda] 132696 512-byte logical blocks (67.9 MB/64.8 MiB) [ 3.785637] vda: detected capacity change from 0 to 67940352 [ 3.800339] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.802314] vdb: detected capacity change from 0 to 1073741824 [ 3.817224] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.820588] vdc: detected capacity change from 0 to 2621440000 [ 3.834644] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.837405] vdd: detected capacity change from 0 to 2621440000 [ 3.851395] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.854452] vde: detected capacity change from 0 to 4294967296 [ 3.868540] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.871320] vdf: detected capacity change from 0 to 4294967296 [ 3.878301] libphy: Fixed MDIO Bus: probed [ 3.888597] usbcore: registered new interface driver usbserial_generic [ 3.890903] usbserial: USB Serial support registered for generic [ 3.893000] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.897124] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.898797] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.901059] mousedev: PS/2 mouse device common for all mice [ 3.905425] rtc_cmos 00:05: RTC can wake from S4 [ 3.906442] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.913390] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.913611] rtc_cmos 00:05: registered as rtc0 [ 3.916914] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.918087] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.923451] intel_pstate: CPU model not supported [ 3.927132] hid: raw HID events driver (C) Jiri Kosina [ 3.929676] usbcore: registered new interface driver usbhid [ 3.931794] usbhid: USB HID core driver [ 3.933590] drop_monitor: Initializing network drop monitor service [ 3.935728] Initializing XFRM netlink socket [ 3.937847] NET: Registered protocol family 10 [ 3.941162] Segment Routing with IPv6 [ 3.942622] NET: Registered protocol family 17 [ 3.947919] mpls_gso: MPLS GSO support [ 3.953430] RAS: Correctable Errors collector initialized. [ 3.955987] AVX version of gcm_enc/dec engaged. [ 3.958239] AES CTR mode by8 optimization enabled [ 4.065588] sched_clock: Marking stable (4065556970, 0)->(4987865068, -922308098) [ 4.070157] registered taskstats version 1 [ 4.072254] Loading compiled-in X.509 certificates [ 4.074440] zswap: loaded using pool lzo/zbud [ 4.105777] Key type big_key registered [ 4.119775] Key type encrypted registered [ 4.121476] ima: No TPM chip found, activating TPM-bypass! [ 4.124108] ima: Allocated hash algorithm: sha1 [ 4.126089] ima: No architecture policies found [ 4.127894] evm: Initialising EVM extended attributes: [ 4.130041] evm: security.selinux [ 4.131392] evm: security.ima [ 4.132426] evm: security.capability [ 4.133508] evm: HMAC attrs: 0x1 [ 4.137747] rtc_cmos 00:05: setting system clock to 2025-07-28 22:41:12 UTC (1753742472) [ 4.145911] debug: unmapping init [mem 0xffffffffa4203000-0xffffffffa43fffff] [ 4.149318] debug: unmapping init [mem 0xffffffffa2f82000-0xffffffffa3258fff] [ 4.162174] Write protecting the kernel read-only data: 28672k [ 4.165339] debug: unmapping init [mem 0xffffffffa1603000-0xffffffffa17fffff] [ 4.168423] debug: unmapping init [mem 0xffffffffa1f14000-0xffffffffa1ffffff] [ 4.215392] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.224248] systemd[1]: Detected virtualization kvm. [ 4.226227] systemd[1]: Detected architecture x86-64. [ 4.228161] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.257473] systemd[1]: No hostname configured. [ 4.259108] systemd[1]: Set hostname to . [ 4.261358] random: systemd: uninitialized urandom read (16 bytes read) [ 4.263706] systemd[1]: Initializing machine ID from random generator. [ 4.314422] random: ln: uninitialized urandom read (6 bytes read) [ 4.401752] random: systemd: uninitialized urandom read (16 bytes read) [ 4.404690] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 4.410283] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.415299] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.074302] device-mapper: uevent: version 1.0.3 [ 5.076942] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [[ 5.765146] random: fast init done  OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.809302] virtio_net virtio0 ens2: renamed from eth0 [ 5.900248] scsi host0: ata_piix [ 5.908701] scsi host1: ata_piix [ 5.910576] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.913558] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.194697] dracut-initqueue[593]: RTNETLINK answers: File exists [ 10.643423] random: crng init done [ 10.645089] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 11.119869] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.377542] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.658961] SELinux: Disabled at runtime. [ 12.724026] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.733300] systemd[1]: Detected virtualization kvm. [ 12.735060] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.318518] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.322264] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.328059] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.332405] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.337401] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.346394] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.351623] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Kernel Socket. Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK [ 13.442286] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS ] Reached target Paths. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target rpc_pipefs.target. Starting Apply Kernel Variables... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... Starting udev Coldplug all Devices... [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.893630] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.312780] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.325067] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.542134] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.562480] EDAC sbridge: Ver: 1.1.2 [ 16.418939] Key type dns_resolver registered [ 16.721347] NFS: Registering the id_resolver key type [ 16.723361] Key type id_resolver registered [ 16.724662] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Crash recovery kernel arming. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg205-server login: [ 43.424295] libcfs: loading out-of-tree module taints kernel. [ 43.437788] Key type ._llcrypt registered [ 43.439345] Key type .llcrypt registered [ 43.490312] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [ 52.569480] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 53.181352] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 53.189923] alg: No test for adler32 (adler32-zlib) [ 54.433730] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 55.106760] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 55.111755] LNet: Accept secure, port 988 [ 56.799282] Key type lgssc registered [ 58.115919] Lustre: Echo OBD driver; http://www.lustre.org/ [ 74.368730] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 77.825485] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 79.552761] hrtimer: interrupt took 11695605 ns [ 91.945804] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 101.643316] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 109.570246] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 124.851237] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 137.700751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 137.748736] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 137.775555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 139.010053] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 139.043819] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 139.138346] Lustre: lustre-MDT0000: new disk, initializing [ 139.247432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 139.271133] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 142.974975] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 156.096695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 156.213677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 156.357952] Lustre: 6460:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 156.416958] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 156.420205] Lustre: Skipped 1 previous similar message [ 156.573227] Lustre: lustre-MDT0001: new disk, initializing [ 156.715409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 156.740547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 156.749737] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 161.138848] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 166.092663] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 175.955731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 176.012417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 176.271835] Lustre: lustre-OST0000: new disk, initializing [ 176.280484] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 176.349939] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 181.757779] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 184.878788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 184.895710] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 184.973788] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 196.165609] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 196.240689] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 196.327962] Lustre: lustre-OST0001: new disk, initializing [ 196.337459] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 196.391579] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 201.337384] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 202.295827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 202.308858] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 202.392062] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 215.211476] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 224.879989] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 236.827737] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing check_logdir /tmp/testlogs/ [ 241.498133] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing yml_node [ 245.571694] Lustre: DEBUG MARKER: Client: 2.16.56.105 [ 248.310563] Lustre: DEBUG MARKER: MDS: 2.16.56.105 [ 250.883635] Lustre: DEBUG MARKER: OSS: 2.16.56.105 [ 252.406976] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Jul 28 18:45:19 EDT 2025 [ 270.862342] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 272.385799] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 289.247960] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 289.247960] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 289.248361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 289.260456] Lustre: Skipped 2 previous similar messages [ 289.291229] Lustre: Skipped 2 previous similar messages [ 293.325790] LustreError: 12128:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 293.530548] Lustre: server umount lustre-MDT0000 complete [ 296.980109] LustreError: 6452:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753742765 with bad export cookie 16850667299217065944 [ 296.981477] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 296.988086] LustreError: 6452:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 299.488559] LustreError: 7697:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 299.488891] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 299.491481] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 299.501080] LustreError: 7697:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 6 previous similar messages [ 299.505466] Lustre: Skipped 1 previous similar message [ 301.538985] LustreError: 6467:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 301.539050] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 301.556742] LustreError: 6467:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 301.563177] Lustre: Skipped 2 previous similar messages [ 303.215402] LustreError: 12330:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 303.217669] LustreError: 12330:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 303.401024] Lustre: server umount lustre-MDT0001 complete [ 306.439757] LustreError: 12532:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 306.447406] LustreError: 12532:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 306.511159] Lustre: server umount lustre-OST0000 complete [ 309.736618] LustreError: 12733:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 309.739485] LustreError: 12733:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 309.901820] Lustre: server umount lustre-OST0001 complete [ 315.881505] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [ 323.913833] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 332.765157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 338.944837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 344.449094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 350.085681] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 357.471539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 357.528314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 357.733908] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 357.796823] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 357.847562] Lustre: lustre-MDT0000: new disk, initializing [ 357.937163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 357.979749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 362.817396] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 374.208376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 374.335500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 374.493407] Lustre: 14904:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 374.605811] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 374.614296] Lustre: Skipped 1 previous similar message [ 374.755276] Lustre: lustre-MDT0001: new disk, initializing [ 374.881396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 374.935193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 374.947724] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 379.518168] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 386.993123] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 391.011935] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 398.340731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 398.438556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 398.765305] Lustre: lustre-OST0000: new disk, initializing [ 398.773088] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 398.877639] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 400.384414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 400.396134] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 400.579859] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 404.530761] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 412.878938] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 416.939560] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 417.150442] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 421.070224] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 421.328562] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 425.956625] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 425.972745] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 426.004480] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 426.464543] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 429.316127] LustreError: 17814:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 429.322793] LustreError: 17814:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 429.473927] Lustre: server umount lustre-OST0000 complete [ 436.203096] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 436.206583] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 436.212617] Lustre: Skipped 1 previous similar message [ 436.229431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 436.232364] Lustre: Skipped 2 previous similar messages [ 439.448819] LustreError: 18016:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 439.461160] LustreError: 18016:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 439.655404] Lustre: server umount lustre-MDT0000 complete [ 441.835032] LustreError: 14911:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 441.845840] LustreError: 14911:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 443.764813] LustreError: 14895:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753742912 with bad export cookie 16850667299217067477 [ 443.769399] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 443.778953] LustreError: 14895:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 444.252870] Lustre: server umount lustre-MDT0001 complete [ 448.404513] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 18:48:35 (1753742915) [ 450.773473] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 452.569547] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 18:48:39 (1753742919) [ 459.241734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 459.620416] LustreError: 18888:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 459.693926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 462.793517] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 464.864207] LustreError: 18888:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 470.376067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 476.292387] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 481.815929] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 485.949285] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 493.756876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 500.662908] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 506.502795] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 515.042606] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 515.046457] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 515.052577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 515.052588] Lustre: Skipped 1 previous similar message [ 515.081934] Lustre: Skipped 2 previous similar messages [ 519.417753] LustreError: 21113:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 519.427468] LustreError: 21113:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 519.573299] Lustre: server umount lustre-OST0000 complete [ 525.283534] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 525.301759] Lustre: Skipped 1 previous similar message [ 530.015376] Lustre: server umount lustre-MDT0000 complete [ 530.438283] LustreError: 19593:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 530.479803] LustreError: 19593:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 534.498692] LustreError: 21317:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753743002 with bad export cookie 16850667299217068359 [ 534.524854] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 535.169877] Lustre: server umount lustre-MDT0001 complete [ 547.586179] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 550.506905] Key type lgssc unregistered [ 550.810612] LNet: 22045:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 550.834238] LNet: Removed LNI 192.168.202.105@tcp [ 551.879719] Key type .llcrypt unregistered [ 551.882481] Key type ._llcrypt unregistered [ 573.484842] Key type ._llcrypt registered [ 573.486574] Key type .llcrypt registered [ 573.616970] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 574.351161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 574.387208] alg: No test for adler32 (adler32-zlib) [ 575.496556] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 575.690181] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 575.694430] LNet: Accept secure, port 988 [ 577.391585] Key type lgssc registered [ 578.519279] Lustre: Echo OBD driver; http://www.lustre.org/ [ 586.925101] loop: module loaded [ 587.008023] loop0: detected capacity change from 0 to 204800000 [ 587.270054] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 587.283983] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 587.547798] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 587.553692] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 587.594394] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 587.636952] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 587.676439] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 587.729441] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 593.587848] loop0: detected capacity change from 0 to 204800000 [ 593.770336] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 593.780289] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 593.808835] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 593.992843] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 599.150107] loop0: detected capacity change from 0 to 204800000 [ 604.101217] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 604.158480] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 604.177154] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 605.476758] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 605.530659] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 605.632513] Lustre: lustre-MDT0000: new disk, initializing [ 605.758916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 605.782553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 609.337403] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 618.303519] loop1: detected capacity change from 0 to 204800000 [ 624.867776] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 624.979158] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 625.455520] Lustre: lustre-OST07c6: new disk, initializing [ 625.458391] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 625.478343] Lustre: Skipped 1 previous similar message [ 625.607964] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 633.552806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 633.573479] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 633.641735] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 634.239978] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 733.503281] LustreError: 26372:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 733.799431] Lustre: server umount lustre-OST07c6 complete [ 754.668044] LustreError: 26873:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 754.671909] LustreError: 26873:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 754.966779] Lustre: server umount lustre-MDT0000 complete [ 776.515635] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 781.640764] Key type lgssc unregistered [ 782.094232] LNet: 27704:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 783.147869] LNet: Removed LNI 192.168.202.105@tcp [ 784.323588] Key type .llcrypt unregistered [ 784.326182] Key type ._llcrypt unregistered [ 805.274307] Key type ._llcrypt registered [ 805.275821] Key type .llcrypt registered [ 805.383990] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 806.381611] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 806.424963] alg: No test for adler32 (adler32-zlib) [ 807.657477] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 808.025443] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 808.038724] LNet: Accept secure, port 988 [ 809.856245] Key type lgssc registered [ 811.361672] Lustre: Echo OBD driver; http://www.lustre.org/ [ 819.920497] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 819.944376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 821.609300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 825.747678] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 830.949219] LustreError: 28945:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 830.971138] LustreError: 28945:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 834.180098] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 834.571371] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 838.145322] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 843.061777] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 846.523321] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 853.087969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 853.626705] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 859.311916] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 864.946562] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 865.900167] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 873.998485] LustreError: 31122:0:(qsd_reint.c:617:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:2 until pending updates are flushed. [ 884.713918] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 884.725894] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 884.744256] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 885.886136] LustreError: 31376:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 886.073628] Lustre: server umount lustre-OST0000 complete [ 890.853094] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 890.872274] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 890.911749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 891.872678] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 897.002584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 897.005789] Lustre: Skipped 2 previous similar messages [ 900.084635] LustreError: 31577:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 900.087632] LustreError: 31577:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 900.159514] Lustre: server umount lustre-MDT0000 complete [ 902.112594] LustreError: 28940:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 902.135548] LustreError: 28940:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 903.902292] LustreError: 28926:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753743372 with bad export cookie 11130665010776750392 [ 903.918570] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 903.923615] LustreError: 28926:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 904.093825] LustreError: 31783:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 904.104047] LustreError: 31783:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 904.377350] Lustre: server umount lustre-MDT0001 complete [ 916.927776] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 919.893253] Key type lgssc unregistered [ 920.192971] LNet: 32309:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 921.258144] LNet: Removed LNI 192.168.202.105@tcp [ 922.222903] Key type .llcrypt unregistered [ 922.229361] Key type ._llcrypt unregistered [ 940.370743] Key type ._llcrypt registered [ 940.372320] Key type .llcrypt registered [ 940.465032] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 941.624722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 941.637597] alg: No test for adler32 (adler32-zlib) [ 942.711791] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 942.935720] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 942.949754] LNet: Accept secure, port 988 [ 944.679124] Key type lgssc registered [ 945.712593] Lustre: Echo OBD driver; http://www.lustre.org/ [ 953.843683] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 953.863726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 955.280720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 958.862770] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 965.602425] LustreError: 33550:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 965.612627] LustreError: 33550:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 966.872481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 967.201788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 970.622254] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 975.775148] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 979.380395] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 986.204485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 986.553441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 991.954184] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 998.419062] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 999.864759] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1015.264018] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1015.274482] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1015.277187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1015.296968] Lustre: Skipped 1 previous similar message [ 1015.349617] Lustre: Skipped 1 previous similar message [ 1018.887223] LustreError: 35801:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1019.160728] Lustre: server umount lustre-OST0000 complete [ 1023.456994] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1023.464897] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1023.479138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1025.504659] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1025.508498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1025.530812] Lustre: Skipped 1 previous similar message [ 1029.562117] LustreError: 36002:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1029.574989] LustreError: 36002:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1029.705553] Lustre: server umount lustre-MDT0000 complete [ 1030.632590] LustreError: 33549:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1030.659898] LustreError: 33549:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 1033.612769] LustreError: 36003:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753743501 with bad export cookie 15240951339024289984 [ 1033.619815] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1033.642617] LustreError: 36003:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1033.794379] LustreError: 36205:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1033.796978] LustreError: 36205:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1034.050118] Lustre: server umount lustre-MDT0001 complete [ 1046.903549] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 1049.706088] Key type lgssc unregistered [ 1050.006524] LNet: 36731:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1051.046382] LNet: Removed LNI 192.168.202.105@tcp [ 1051.807359] Key type .llcrypt unregistered [ 1051.810484] Key type ._llcrypt unregistered [ 1074.297444] Key type ._llcrypt registered [ 1074.299092] Key type .llcrypt registered [ 1074.383328] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 1076.176321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1076.248560] alg: No test for adler32 (adler32-zlib) [ 1077.330270] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 1077.508312] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 1077.512296] LNet: Accept secure, port 988 [ 1079.175227] Key type lgssc registered [ 1080.890144] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1090.468983] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1090.498129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1091.898956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1095.852835] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1102.308957] LustreError: 38124:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1102.337820] LustreError: 38124:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1105.541728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1105.842829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1110.022786] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1115.791880] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1120.273679] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1128.515222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1128.852903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1135.590640] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1137.084062] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1141.710341] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1149.785709] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.5@tcp (stopping) [ 1151.971211] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1151.986318] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1152.001869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1154.496954] LustreError: 40355:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1154.649350] Lustre: server umount lustre-MDT0000 complete [ 1154.872097] LustreError: 38118:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1157.601687] LustreError: 38118:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1157.612038] LustreError: 38118:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1158.265762] LustreError: 38104:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753743626 with bad export cookie 12776946863411791312 [ 1158.273220] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1158.277247] LustreError: 38104:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1159.983270] LustreError: 38118:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1160.037260] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.5@tcp (stopping) [ 1160.050944] Lustre: Skipped 3 previous similar messages [ 1162.722038] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1162.748829] Lustre: Skipped 2 previous similar messages [ 1162.752216] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1164.464613] LustreError: 40556:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1164.473214] LustreError: 40556:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1164.741837] Lustre: server umount lustre-MDT0001 complete [ 1185.596395] LustreError: 40779:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1185.606321] LustreError: 40779:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1185.848895] Lustre: server umount lustre-OST0000 complete [ 1207.286844] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 1210.825902] Key type lgssc unregistered [ 1211.222521] LNet: 41625:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1212.261881] LNet: Removed LNI 192.168.202.105@tcp [ 1213.406424] Key type .llcrypt unregistered [ 1213.410289] Key type ._llcrypt unregistered [ 1235.662996] Key type ._llcrypt registered [ 1235.664777] Key type .llcrypt registered [ 1235.776824] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 1236.898629] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1236.928893] alg: No test for adler32 (adler32-zlib) [ 1238.019650] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 1238.313649] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 1238.318359] LNet: Accept secure, port 988 [ 1239.991555] Key type lgssc registered [ 1241.022579] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1247.642199] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1247.667224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1249.069110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1252.047094] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1259.488690] LustreError: 43013:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1259.504143] LustreError: 43013:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1259.715278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1260.185028] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1263.766279] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1269.505181] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1273.139621] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1279.660596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1279.998176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1285.780288] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1291.478237] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1296.373939] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1301.473557] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1301.479103] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1301.495995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1302.349099] Lustre: lustre-OST0000: Not available for connect from 192.168.202.5@tcp (stopping) [ 1302.353482] Lustre: Skipped 1 previous similar message [ 1306.596783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1307.269186] LustreError: 45241:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1307.415551] Lustre: server umount lustre-OST0000 complete [ 1326.561932] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1326.571808] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1326.590902] Lustre: Skipped 1 previous similar message [ 1326.594433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1326.599456] Lustre: Skipped 1 previous similar message [ 1331.899612] LustreError: 45616:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1331.902570] LustreError: 45616:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1332.085840] Lustre: server umount lustre-MDT0000 complete [ 1332.202350] LustreError: 43012:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1332.223429] LustreError: 43012:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 1336.018359] LustreError: 42994:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753743804 with bad export cookie 10332972622438732804 [ 1336.023979] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1336.033125] LustreError: 42994:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1342.304253] LustreError: 45820:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1342.318106] LustreError: 45820:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1342.603526] Lustre: server umount lustre-MDT0001 complete [ 1356.007343] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 1359.164163] Key type lgssc unregistered [ 1359.513617] LNet: 46349:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1360.560344] LNet: Removed LNI 192.168.202.105@tcp [ 1361.603039] Key type .llcrypt unregistered [ 1361.605208] Key type ._llcrypt unregistered [ 1389.527517] Key type ._llcrypt registered [ 1389.529254] Key type .llcrypt registered [ 1389.672492] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 1391.021798] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1391.049237] alg: No test for adler32 (adler32-zlib) [ 1392.165978] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 1392.383283] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 1392.391137] LNet: Accept secure, port 988 [ 1394.111425] Key type lgssc registered [ 1395.244311] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1402.934449] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1402.967519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1404.570354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1408.751160] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1414.113470] LustreError: 47738:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1414.127483] LustreError: 47738:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1416.372703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1416.708566] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1420.881652] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1425.886126] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1429.348831] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1435.922717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1436.310434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1441.527821] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1447.350804] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1449.657457] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1458.911500] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1463.013700] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1468.907255] LNetError: Refusing connection from 192.168.202.105 for 127.0.0.2@tcp: No matching NI [ 1468.912466] LNetError: 47041:0:(socklnd_cb.c:1794:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1468.918936] LNetError: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1475.266905] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2025-07-2819h05m41s [ 1476.985067] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1479.998038] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.5@tcp (stopping) [ 1480.672511] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1480.674708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1480.680796] LNetError: Refusing connection from 192.168.202.105 for 127.0.0.2@tcp: No matching NI [ 1480.680870] LNetError: 47044:0:(socklnd_cb.c:1794:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1480.680895] LNetError: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1480.687923] Lustre: Skipped 2 previous similar messages [ 1480.739019] Lustre: Skipped 2 previous similar messages [ 1481.762456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1481.765274] Lustre: Skipped 3 previous similar messages [ 1485.340302] LustreError: 50298:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1485.590184] Lustre: server umount lustre-MDT0000 complete [ 1486.133541] LustreError: 49834:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1486.824607] LNetError: Refusing connection from 192.168.202.105 for 127.0.0.2@tcp: No matching NI [ 1486.827439] LNetError: 47042:0:(socklnd_cb.c:1794:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1486.831188] LNetError: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1487.908086] LustreError: 47734:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1487.936794] LustreError: 47734:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1489.205932] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1492.269432] LustreError: 47734:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1492.965916] LNetError: Refusing connection from 192.168.202.105 for 127.0.0.2@tcp: No matching NI [ 1492.968500] LNetError: 47043:0:(socklnd_cb.c:1794:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1492.987719] LNetError: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1496.158136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1496.236650] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1496.494203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1496.550310] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1497.515360] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 1499.689253] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1503.106624] Lustre: DEBUG MARKER: Wait for df (36052) ... [ 1505.205501] Lustre: DEBUG MARKER: done [ 1511.905777] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1511.913830] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1511.927328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1512.936523] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1518.051258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1518.092588] Lustre: Skipped 3 previous similar messages [ 1523.168447] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1523.217257] LustreError: 51349:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1523.233833] LustreError: 51349:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1523.481130] Lustre: server umount lustre-OST0000 complete [ 1528.289806] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1528.293154] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1528.300554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1528.300562] Lustre: Skipped 2 previous similar messages [ 1528.361170] Lustre: Skipped 1 previous similar message [ 1528.373167] LNetError: Refusing connection from 192.168.202.105 for 127.0.0.2@tcp: No matching NI [ 1528.393587] LNetError: 47041:0:(socklnd_cb.c:1794:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1528.407164] LNetError: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1533.521313] LustreError: 51552:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1533.529809] LustreError: 51552:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1533.689740] Lustre: server umount lustre-MDT0000 complete [ 1534.435076] LustreError: 47737:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1534.454628] LustreError: 47737:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 1536.941198] LustreError: 50028:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753744005 with bad export cookie 597347109252422463 [ 1536.945263] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1536.962710] LustreError: 50028:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1537.077146] LustreError: 51753:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1537.084991] LustreError: 51753:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1537.441054] Lustre: server umount lustre-MDT0001 complete [ 1550.364615] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 1553.306764] Key type lgssc unregistered [ 1553.616406] LNet: 52280:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1554.666571] LNet: Removed LNI 192.168.202.105@tcp [ 1555.581665] Key type .llcrypt unregistered [ 1555.583470] Key type ._llcrypt unregistered [ 1590.727774] Key type ._llcrypt registered [ 1590.730376] Key type .llcrypt registered [ 1590.837548] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [ 1604.722813] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 1605.577343] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1605.665058] alg: No test for adler32 (adler32-zlib) [ 1606.776814] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 1607.020824] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 1607.025755] LNet: Accept secure, port 988 [ 1608.695168] Key type lgssc registered [ 1609.839445] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1619.106131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1625.788622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1632.536914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1639.549390] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1645.796198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1645.881765] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1645.910819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1647.154821] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1647.216875] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1647.342462] Lustre: lustre-MDT0000: new disk, initializing [ 1647.439639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1647.466701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1650.999832] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1660.907126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1660.980718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1661.092266] Lustre: 56224:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1661.155665] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1661.158218] Lustre: Skipped 1 previous similar message [ 1661.283887] Lustre: lustre-MDT0001: new disk, initializing [ 1661.424242] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1661.469179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1661.486215] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1665.505320] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1672.312278] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1676.267555] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1683.782945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1683.867049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1684.149590] Lustre: lustre-OST0000: new disk, initializing [ 1684.155843] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1684.237607] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1689.703703] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1692.798195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1692.808371] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1692.932381] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1697.461032] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1701.483550] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1701.732045] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1705.278856] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1705.517819] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1708.516806] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1708.517121] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1708.524601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1708.532643] Lustre: Skipped 1 previous similar message [ 1708.543230] Lustre: Skipped 1 previous similar message [ 1713.360212] LustreError: 59130:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1713.532817] Lustre: server umount lustre-OST0000 complete [ 1717.728968] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1717.732844] LustreError: Skipped 1 previous similar message [ 1717.735347] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1717.742610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1718.754413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1718.761659] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1723.236315] LustreError: 59331:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1723.245978] LustreError: 59331:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1723.367550] Lustre: server umount lustre-MDT0000 complete [ 1726.452619] LustreError: 56215:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753744194 with bad export cookie 2608805071747296272 [ 1726.458214] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1726.484341] LustreError: 56215:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1726.609768] LustreError: 59532:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1726.613992] LustreError: 59532:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1726.860177] Lustre: server umount lustre-MDT0001 complete [ 1735.185552] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 19:10:01 (1753744201) [ 1743.117774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1743.743137] LustreError: 60205:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1743.761375] LustreError: 60205:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 1743.848256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1747.899111] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1748.967350] LustreError: 60205:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1754.083679] LustreError: 60205:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1756.676529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1756.960312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1760.833461] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1767.208792] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1770.700604] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1777.515162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1777.766663] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1783.413701] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1789.174154] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1795.581583] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2025-07-2819h11m02s [ 1797.123327] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1803.979119] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1804.795028] LNetError: Refusing connection from 192.168.202.105 for 127.0.0.2@tcp: No matching NI [ 1804.811104] LNetError: 54426:0:(socklnd_cb.c:1794:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1804.824834] LNetError: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1806.371679] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1806.426990] LustreError: 60199:0:(mdt_handler.c:2329:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1826.641659] Lustre: lustre-MDT0000: Client 4cddead5-ffac-4ec5-899f-a7154f3f121b (at 192.168.202.5@tcp) reconnecting [ 1846.513432] LustreError: 60199:0:(mdt_handler.c:2329:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1848.335930] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1850.604296] Lustre: DEBUG MARKER: done [ 1855.972550] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1855.975838] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1855.990799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1855.992664] Lustre: Skipped 1 previous similar message [ 1855.994718] Lustre: Skipped 1 previous similar message [ 1861.099567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1861.492297] LustreError: 63192:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1861.511869] LustreError: 63192:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1861.774082] Lustre: server umount lustre-OST0000 complete [ 1866.210943] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1866.221572] Lustre: Skipped 1 previous similar message [ 1866.233761] LNetError: Refusing connection from 192.168.202.105 for 127.0.0.2@tcp: No matching NI [ 1866.246697] LNetError: 54429:0:(socklnd_cb.c:1794:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1866.266953] LNetError: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1871.328489] LNetError: Refusing connection from 192.168.202.105 for 127.0.0.2@tcp: No matching NI [ 1871.332634] LNetError: 54427:0:(socklnd_cb.c:1794:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1871.337120] LNetError: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1871.577863] LustreError: 63394:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1871.583548] LustreError: 63394:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1871.712401] Lustre: server umount lustre-MDT0000 complete [ 1872.352938] LustreError: 62886:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1872.379015] LustreError: 62886:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 1874.991376] LustreError: 60185:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753744343 with bad export cookie 2608805071747297154 [ 1874.996821] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1875.003093] LustreError: 60185:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1875.300482] Lustre: server umount lustre-MDT0001 complete [ 1885.515515] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 1887.611963] Key type lgssc unregistered [ 1887.856614] LNet: 64123:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1888.869841] LNet: Removed LNI 192.168.202.105@tcp [ 1889.529863] Key type .llcrypt unregistered [ 1889.532624] Key type ._llcrypt unregistered [ 1917.489225] Key type ._llcrypt registered [ 1917.490859] Key type .llcrypt registered [ 1917.593622] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [ 1928.622211] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 1929.197264] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1929.220690] alg: No test for adler32 (adler32-zlib) [ 1930.127131] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 1930.284214] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 1930.286930] LNet: Accept secure, port 988 [ 1931.927184] Key type lgssc registered [ 1932.785892] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1939.766079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1945.109641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1950.610246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1956.561293] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1962.173319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1962.219130] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1962.228856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1963.389271] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1963.425858] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1963.500396] Lustre: lustre-MDT0000: new disk, initializing [ 1963.570715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1963.606867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1966.538541] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1975.029942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1975.096423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1975.197654] Lustre: 68038:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1975.251913] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1975.255588] Lustre: Skipped 1 previous similar message [ 1975.313677] Lustre: lustre-MDT0001: new disk, initializing [ 1975.420533] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1975.453712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1975.463705] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1978.834404] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1984.750366] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1987.610260] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1992.572749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1992.619833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1992.786631] Lustre: lustre-OST0000: new disk, initializing [ 1992.790857] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1992.848587] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1997.064155] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2002.467278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2002.494085] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2002.573904] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2003.850398] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2008.106121] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2008.393715] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2013.195540] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2013.512169] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2018.279251] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2018.279649] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2018.303342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2018.306968] LustreError: Skipped 1 previous similar message [ 2018.315070] Lustre: Skipped 1 previous similar message [ 2021.577257] LustreError: 70944:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2021.883420] Lustre: server umount lustre-OST0000 complete [ 2026.468165] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2026.486227] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2026.518932] Lustre: Skipped 1 previous similar message [ 2026.536756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2028.514299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2028.517851] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2032.342936] LustreError: 71145:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2032.346120] LustreError: 71145:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2032.436973] Lustre: server umount lustre-MDT0000 complete [ 2035.840726] LustreError: 68028:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753744504 with bad export cookie 14306302636165189198 [ 2035.852636] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2035.867046] LustreError: 68028:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2035.994862] LustreError: 71347:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2035.997121] LustreError: 71347:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2036.128302] Lustre: server umount lustre-MDT0001 complete [ 2044.402423] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 19:15:11 (1753744511) [ 2046.194330] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 2048.114391] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 19:15:14 (1753744514) [ 2058.098733] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 2061.634565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2062.031966] LustreError: 72385:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2062.045272] LustreError: 72385:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 2062.086284] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2063.065112] Lustre: Failing over lustre-MDT0000 [ 2063.078189] LustreError: 72413:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2063.099547] LustreError: 72413:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2063.107839] LustreError: 72413:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 1, retries 0, failed: rc = -5 [ 2063.288939] LustreError: 72464:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2063.291455] LustreError: 72464:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2063.618155] Lustre: server umount lustre-MDT0000 complete [ 2071.054471] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 19:15:37 (1753744537) [ 2078.240446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2078.779221] LustreError: 73033:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2078.873511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2082.860746] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2084.320764] LustreError: 73033:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2089.453597] LustreError: 73032:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2090.650858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2091.056346] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2094.642472] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2099.860873] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2103.288473] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2109.125757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2109.398952] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2114.474436] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2118.792143] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2127.947201] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2133.987305] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2133.989359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2133.998154] Lustre: Skipped 1 previous similar message [ 2134.006886] Lustre: Skipped 3 previous similar messages [ 2135.073854] LustreError: 75358:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2135.082693] LustreError: 75358:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2135.229772] Lustre: server umount lustre-MDT0000 complete [ 2138.996214] LustreError: 73746:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753744607 with bad export cookie 14306302636165190360 [ 2138.998633] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2139.009086] LustreError: 73746:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2139.105130] LustreError: 73027:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2139.105240] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2139.146721] Lustre: Skipped 1 previous similar message [ 2139.150938] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2139.437249] Lustre: server umount lustre-MDT0001 complete [ 2143.168058] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2147.500232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2156.338893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2164.961873] LustreError: 74629:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2165.047920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2165.156201] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2169.609379] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2177.392984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2178.666029] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 2181.025117] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2186.036712] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2187.881738] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 2189.233988] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2196.446762] LustreError: 77807:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2196.451891] LustreError: 77807:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2196.567124] Lustre: server umount lustre-MDT0000 complete [ 2197.984630] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2197.992159] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2198.002588] LustreError: 77132:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2198.013938] LustreError: 77132:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 2199.815322] LustreError: 73012:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753744668 with bad export cookie 14306302636165192264 [ 2199.817494] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2199.833763] LustreError: 73012:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2200.102920] Lustre: server umount lustre-MDT0001 complete [ 2205.082698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2213.215641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2218.984156] Lustre: 66254:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1753744671/real 1753744671] req@0000000051ec1573 x1838934285969920/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1753744687 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2219.010292] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2219.020680] Lustre: Skipped 2 previous similar messages [ 2220.770255] Lustre: 66255:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1753744673/real 1753744673] req@0000000058382271 x1838934285970048/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1753744689 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2225.119107] Lustre: 66253:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1753744677/real 1753744677] req@000000004534f525 x1838934285970176/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1753744693 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2225.185500] Lustre: Evicted from MGS (at 192.168.202.105@tcp) after server handle changed from 0x0 to 0xc68a327315dc5fa4 [ 2225.205667] Lustre: MGC192.168.202.105@tcp: Connection restored to (at 0@lo) [ 2225.541343] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2228.288991] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2231.269157] Lustre: 66254:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1753744683/real 1753744683] req@0000000014ffb110 x1838934285970560/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1753744699 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2231.283325] Lustre: 66254:0:(client.c:2453:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2233.946396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2234.156256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2234.159865] Lustre: Skipped 2 previous similar messages [ 2237.199623] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2238.246872] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 2241.507585] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2244.147429] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2248.676250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2248.679019] Lustre: Skipped 2 previous similar messages [ 2254.813632] LustreError: 80256:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2254.818386] LustreError: 80256:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2254.919536] Lustre: server umount lustre-MDT0000 complete [ 2257.888968] LustreError: 77809:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753744726 with bad export cookie 14306302636165193636 [ 2257.890044] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2257.932778] LustreError: 77809:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2258.276440] Lustre: server umount lustre-MDT0001 complete [ 2263.165915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2266.019096] Lustre: DEBUG MARKER: files compared the same [ 2276.827429] Lustre: server umount lustre-OST0000 complete [ 2290.598705] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 2293.435960] Key type lgssc unregistered [ 2293.685672] LNet: 81884:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2294.759724] LNet: Removed LNI 192.168.202.105@tcp [ 2295.468909] Key type .llcrypt unregistered [ 2295.475667] Key type ._llcrypt unregistered [ 2311.901204] Key type ._llcrypt registered [ 2311.903327] Key type .llcrypt registered [ 2311.968181] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 2312.450612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2312.478222] alg: No test for adler32 (adler32-zlib) [ 2313.393291] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 2313.521099] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 2313.528810] LNet: Accept secure, port 988 [ 2315.143458] Key type lgssc registered [ 2315.692677] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2320.481488] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2320.503634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2321.938632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2324.774184] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug +malloc all [ 2330.488569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2330.716290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2333.685263] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug +malloc all [ 2337.629267] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2340.515780] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2346.775427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2347.000113] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2348.036089] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2351.406850] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug +malloc all [ 2355.337242] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2363.365052] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2363.371927] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2363.382581] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2366.431716] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2366.434063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2366.437511] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2366.457075] Lustre: Skipped 1 previous similar message [ 2368.566178] LustreError: 85472:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2368.678988] Lustre: server umount lustre-OST0000 complete [ 2371.552349] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2371.553282] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2371.562106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2371.566713] Lustre: Skipped 1 previous similar message [ 2376.673814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2376.678042] Lustre: Skipped 2 previous similar messages [ 2377.701776] LustreError: 85677:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2377.714495] LustreError: 85677:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2377.813922] Lustre: server umount lustre-MDT0000 complete [ 2380.877545] LustreError: 83231:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753744849 with bad export cookie 12787395966819744403 [ 2380.881898] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2380.886491] LustreError: 83231:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2380.966777] LustreError: 85879:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2380.973084] LustreError: 85879:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2381.109275] Lustre: server umount lustre-MDT0001 complete [ 2389.294069] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 2391.470091] Key type lgssc unregistered [ 2391.706346] LNet: 86406:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2392.748331] LNet: Removed LNI 192.168.202.105@tcp [ 2393.330202] Key type .llcrypt unregistered [ 2393.331482] Key type ._llcrypt unregistered [ 2411.373636] Key type ._llcrypt registered [ 2411.375582] Key type .llcrypt registered [ 2411.439874] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 2412.203680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2412.219480] alg: No test for adler32 (adler32-zlib) [ 2413.210580] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 2413.367687] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 2413.372653] LNet: Accept secure, port 988 [ 2415.023430] Key type lgssc registered [ 2415.922857] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2422.307136] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2422.323700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2439.843828] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2445.983231] LustreError: 87743:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.105@tcp: failed processing log, type 4: rc = -110 [ 2475.743236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2480.796414] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2485.117554] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2490.818075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2490.978811] Lustre: *** cfs_fail_loc=706, val=0*** [ 2491.323397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2491.405804] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2494.231326] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2496.489739] LustreError: 87770:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2496.497046] LustreError: 87770:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 2501.089127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2501.412659] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2504.543443] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2509.113844] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2512.200574] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2516.961728] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2516.966750] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2516.969021] LustreError: Skipped 1 previous similar message [ 2516.973223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2516.976811] Lustre: Skipped 1 previous similar message [ 2516.997266] Lustre: Skipped 1 previous similar message [ 2519.014135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2519.017201] Lustre: Skipped 1 previous similar message [ 2520.116282] LustreError: 90052:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 2520.232549] Lustre: server umount lustre-OST0000 complete [ 2524.133610] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2524.141197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2529.251435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2529.259711] Lustre: Skipped 2 previous similar messages [ 2529.711423] LustreError: 90254:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2529.723615] LustreError: 90254:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2529.866328] Lustre: server umount lustre-MDT0000 complete [ 2533.234383] LustreError: 89432:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753745001 with bad export cookie 18251634940139190510 [ 2533.242086] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2533.371066] LustreError: 90456:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2533.432598] LustreError: 90456:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2533.682588] Lustre: server umount lustre-MDT0001 complete [ 2542.525982] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 2544.952526] Key type lgssc unregistered [ 2545.243598] LNet: 90983:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2546.280957] LNet: Removed LNI 192.168.202.105@tcp [ 2547.105093] Key type .llcrypt unregistered [ 2547.106664] Key type ._llcrypt unregistered [ 2569.241400] Key type ._llcrypt registered [ 2569.243194] Key type .llcrypt registered [ 2569.339460] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 2570.351553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2570.388834] alg: No test for adler32 (adler32-zlib) [ 2571.406355] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 2571.592634] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 2571.595431] LNet: Accept secure, port 988 [ 2573.271141] Key type lgssc registered [ 2574.304545] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2580.857239] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2580.894919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2584.838802] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2590.662209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2591.041280] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2594.258689] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2596.321130] LustreError: 93026:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2596.330104] LustreError: 93026:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 2601.315369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2601.444497] LustreError: 93030:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2601.456176] LustreError: 93030:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 2601.585033] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2606.145643] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2607.076913] LustreError: 93025:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2607.094355] LustreError: 93025:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 2612.199456] LustreError: 93029:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2613.851918] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2613.860711] Lustre: MGS: Client c49029de-3c9c-464a-a62f-ac198f8b5872 (at 0@lo) reconnecting [ 2614.074609] LustreError: 93040:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2614.079228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2614.155524] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2617.528179] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2629.603751] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2629.604351] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2629.611464] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2629.619073] Lustre: Skipped 1 previous similar message [ 2634.730921] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2634.734034] Lustre: Skipped 1 previous similar message [ 2635.131831] LustreError: 94969:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2635.273096] Lustre: server umount lustre-OST0000 complete [ 2639.842860] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2639.842960] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2639.858675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2639.886817] Lustre: Skipped 1 previous similar message [ 2644.198534] LustreError: 95171:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2644.208244] LustreError: 95171:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2644.241086] Lustre: server umount lustre-MDT0000 complete [ 2644.965445] LustreError: 93030:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2644.985729] LustreError: 93030:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 2647.889846] LustreError: 95372:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2647.896499] LustreError: 95372:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2648.103995] Lustre: server umount lustre-MDT0001 complete [ 2651.685604] LustreError: 95572:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2651.688500] LustreError: 95572:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2651.758270] Lustre: server umount MGS complete [ 2660.788944] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 19:25:27 (1753745127) [ 2685.360683] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [ 2693.295111] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 2702.182640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2707.920035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2713.006207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2718.508404] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2724.709033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2724.760484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2728.290519] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2737.313220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2737.386098] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2737.603110] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2737.644349] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2737.727853] Lustre: lustre-MDT0001: new disk, initializing [ 2737.822451] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2737.854990] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2737.861405] Lustre: 100066:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2737.868014] LustreError: 100066:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2741.512477] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2750.370065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2750.440147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2750.734273] Lustre: lustre-OST0000: new disk, initializing [ 2750.737259] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2750.789430] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2753.825677] Lustre: 100949:0:(client.c:1611:after_reply()) @@@ resending request on EINPROGRESS req@00000000b51847ea x1838934958054016/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1753745238 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2755.425550] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2761.481377] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2767.259638] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2767.266408] Lustre: MGS: Client bad325c1-5560-4fae-8ab5-600adb42a7c1 (at 0@lo) reconnecting [ 2767.290577] Lustre: 99357:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2767.304166] LustreError: 99357:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 2767.327313] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2767.386594] Lustre: lustre-MDT0000: new disk, initializing [ 2767.532625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2767.558293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2767.639137] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2768.613306] Lustre: 101766:0:(client.c:1611:after_reply()) @@@ resending request on EINPROGRESS req@00000000f2d2f0bf x1838934958060416/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1753745252 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2770.440412] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2773.740860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2773.745122] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2773.775728] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 2799.585253] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2799.591622] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2799.602359] Lustre: Skipped 1 previous similar message [ 2799.608129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2804.598553] LustreError: 102285:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 2804.608618] LustreError: 102285:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2804.840163] Lustre: server umount lustre-OST0000 complete [ 2808.288458] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2808.290072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2808.312298] Lustre: Skipped 2 previous similar messages [ 2808.314788] Lustre: Skipped 3 previous similar messages [ 2814.500786] LustreError: 102487:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 2814.509215] LustreError: 102487:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2814.542618] Lustre: server umount lustre-MDT0000 complete [ 2818.530399] LustreError: 100036:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2818.552859] LustreError: 100036:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 2825.017324] Lustre: server umount lustre-MDT0001 complete [ 2827.952406] Lustre: server umount MGS complete [ 2839.478184] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 19:28:26 (1753745306) [ 2855.278844] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 2857.956643] Key type lgssc unregistered [ 2858.221194] LNet: 104222:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2859.237386] LNet: Removed LNI 192.168.202.105@tcp [ 2860.048060] Key type .llcrypt unregistered [ 2860.050979] Key type ._llcrypt unregistered [ 2874.446187] Key type ._llcrypt registered [ 2874.447756] Key type .llcrypt registered [ 2874.526536] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 2875.260324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2875.315101] alg: No test for adler32 (adler32-zlib) [ 2876.395707] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 2876.606666] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 2876.612186] LNet: Accept secure, port 988 [ 2878.255180] Key type lgssc registered [ 2879.091908] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2885.568134] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2885.592924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2887.005870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2890.607122] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2897.384632] LustreError: 105411:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2897.412795] LustreError: 105411:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 2899.087503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2899.512868] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2903.136441] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2908.993924] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2912.916725] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2919.616174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2919.938687] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2925.428869] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2931.299426] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2932.220350] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:33) [ 2942.435948] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2942.438086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2942.467733] Lustre: Skipped 1 previous similar message [ 2945.531929] LustreError: 107640:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2945.745858] Lustre: server umount lustre-OST0000 complete [ 2950.631682] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2950.635095] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2950.664110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2950.667846] Lustre: Skipped 1 previous similar message [ 2952.673370] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2952.677831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2952.704028] Lustre: Skipped 1 previous similar message [ 2956.044503] LustreError: 107842:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2956.047484] LustreError: 107842:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2956.236746] Lustre: server umount lustre-MDT0000 complete [ 2957.794532] LustreError: 105411:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2957.808164] LustreError: 105411:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 2959.713294] LustreError: 105392:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753745428 with bad export cookie 10535191359651459179 [ 2959.718250] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2959.727963] LustreError: 105392:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2959.889105] LustreError: 108043:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2959.894192] LustreError: 108043:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2960.308702] Lustre: server umount lustre-MDT0001 complete [ 2971.152831] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 2973.621612] Key type lgssc unregistered [ 2973.896286] LNet: 108569:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2974.951213] LNet: Removed LNI 192.168.202.105@tcp [ 2975.719348] Key type .llcrypt unregistered [ 2975.722173] Key type ._llcrypt unregistered [ 2991.083226] Key type ._llcrypt registered [ 2991.084427] Key type .llcrypt registered [ 2991.205339] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 2992.088077] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2992.136670] alg: No test for adler32 (adler32-zlib) [ 2993.152861] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 2993.442430] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 2993.449446] LNet: Accept secure, port 988 [ 2995.135225] Key type lgssc registered [ 2996.313784] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3002.146134] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3002.150146] LustreError: 109616:0:(tgt_mount.c:2228:server_fill_super()) cfs_race id 716 sleeping [ 3002.177916] LustreError: 109624:0:(tgt_mount.c:2228:server_fill_super()) cfs_fail_race id 716 waking [ 3002.182965] LustreError: 109616:0:(tgt_mount.c:2228:server_fill_super()) cfs_fail_race id 716 awake: rc=4975 [ 3002.186344] LustreError: 109616:0:(tgt_mount.c:2240:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3002.195047] LustreError: 109616:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3002.217448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3003.656933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3011.484960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3011.814910] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3015.461231] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3019.494917] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing lsmod [ 3021.857226] LustreError: 110779:0:(tgt_mount.c:2228:server_fill_super()) cfs_race id 716 sleeping [ 3021.891166] LustreError: 110790:0:(tgt_mount.c:2228:server_fill_super()) cfs_fail_race id 716 waking [ 3021.895500] LustreError: 110779:0:(tgt_mount.c:2228:server_fill_super()) cfs_fail_race id 716 awake: rc=4966 [ 3021.902789] LustreError: 110779:0:(tgt_mount.c:2240:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3021.906787] LustreError: 110779:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3021.917129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3021.999363] LustreError: 110802:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3022.013811] LustreError: 110802:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3022.167964] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3027.430068] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3027.433558] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3027.438974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3027.442780] Lustre: Skipped 1 previous similar message [ 3030.794759] LustreError: 110979:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3030.944150] Lustre: server umount lustre-MDT0000 complete [ 3032.551523] LustreError: 110164:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3032.570972] LustreError: 110164:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3034.785688] LustreError: 109639:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753745503 with bad export cookie 10883281006919441865 [ 3034.794780] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3034.807306] LustreError: 109639:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3034.978408] LustreError: 111182:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3034.981987] LustreError: 111182:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3035.244774] Lustre: server umount lustre-MDT0001 complete [ 3049.094276] LustreError: 111382:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3049.112732] LustreError: 111382:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3049.304387] Lustre: server umount lustre-OST0000 complete [ 3056.882053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3057.335597] LustreError: 111860:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3057.411458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3061.084719] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3062.757813] LustreError: 111860:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3066.852082] LustreError: 111860:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3068.750721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3069.071572] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3072.754303] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3077.560942] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3081.399322] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3087.968834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3088.190715] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3094.190963] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3099.922477] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3102.517680] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:65) [ 3110.880123] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3110.890386] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3110.917922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3110.920499] Lustre: Skipped 2 previous similar messages [ 3112.928276] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3112.932592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3112.936774] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3112.958134] Lustre: Skipped 1 previous similar message [ 3116.042209] LustreError: 114086:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3116.053201] LustreError: 114086:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3116.258595] Lustre: server umount lustre-OST0000 complete [ 3120.609027] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3120.615106] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3120.638962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3126.177500] LustreError: 114288:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3126.180420] LustreError: 114288:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3126.240363] Lustre: server umount lustre-MDT0000 complete [ 3128.296846] LustreError: 111860:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3128.317664] LustreError: 111860:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 3129.745769] LustreError: 111841:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753745598 with bad export cookie 10883281006919442579 [ 3129.751766] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3129.758867] LustreError: 111841:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3130.086343] Lustre: server umount lustre-MDT0001 complete [ 3140.452734] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 3143.156798] Key type lgssc unregistered [ 3143.532329] LNet: 115018:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3144.614889] LNet: Removed LNI 192.168.202.105@tcp [ 3145.356472] Key type .llcrypt unregistered [ 3145.358219] Key type ._llcrypt unregistered [ 3166.477903] Key type ._llcrypt registered [ 3166.479588] Key type .llcrypt registered [ 3166.580591] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 3167.470217] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3167.527369] alg: No test for adler32 (adler32-zlib) [ 3168.583583] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 3168.792840] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 3168.795617] LNet: Accept secure, port 988 [ 3170.439183] Key type lgssc registered [ 3171.513150] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3179.049561] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3179.064588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3180.565896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3184.107616] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3189.728648] LustreError: 116408:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3189.757835] LustreError: 116408:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3193.323371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3193.703036] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3197.132506] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3202.274739] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3206.074765] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3212.886092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3213.152808] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3217.396440] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3222.345831] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3225.454607] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:67 to 0x240000400:97) [ 3230.730184] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3233.314931] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3238.079573] LustreError: 118737:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3238.198327] Lustre: server umount lustre-OST0000 complete [ 3240.931467] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3240.934271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3240.940581] Lustre: Skipped 1 previous similar message [ 3246.049617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3246.057880] Lustre: Skipped 2 previous similar messages [ 3251.173569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3255.071499] Lustre: 115723:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1753745707/real 1753745707] req@00000000c0a9e89e x1838935585091456/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1753745723 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3255.091421] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3255.100676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3255.103438] Lustre: Skipped 1 previous similar message [ 3255.263132] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3255.346794] LustreError: 118943:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3255.357523] LustreError: 118943:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3255.424178] Lustre: server umount lustre-MDT0000 complete [ 3257.183119] Lustre: 115722:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1753745709/real 1753745709] req@0000000017359825 x1838935585095040/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1753745725 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3258.231972] LustreError: 116389:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753745726 with bad export cookie 9636528791331807132 [ 3258.240912] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3258.243750] LustreError: 116389:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3258.373432] LustreError: 119145:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3258.376231] LustreError: 119145:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3258.571512] Lustre: server umount lustre-MDT0001 complete [ 3268.200666] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 3270.787132] Key type lgssc unregistered [ 3271.055853] LNet: 119673:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3272.102702] LNet: Removed LNI 192.168.202.105@tcp [ 3272.775101] Key type .llcrypt unregistered [ 3272.777427] Key type ._llcrypt unregistered [ 3288.101102] Key type ._llcrypt registered [ 3288.102564] Key type .llcrypt registered [ 3288.210879] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 3289.018536] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3289.070611] alg: No test for adler32 (adler32-zlib) [ 3290.061345] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 3290.241139] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 3290.245388] LNet: Accept secure, port 988 [ 3291.881514] Key type lgssc registered [ 3292.771775] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3298.316402] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3298.333201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3299.664269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3302.445534] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3308.303248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3308.742442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3312.089818] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3316.603159] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3319.333744] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3324.436959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3324.726055] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3325.746071] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 3328.921457] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3333.488260] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3341.284212] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3341.287694] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3341.298300] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3343.328416] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3343.339258] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3343.342025] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3346.164140] LustreError: 123090:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3346.280391] Lustre: server umount lustre-OST0000 complete [ 3349.986840] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3349.999283] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3350.013587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3350.017687] Lustre: Skipped 1 previous similar message [ 3353.569731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3353.574960] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3353.585483] Lustre: Skipped 1 previous similar message [ 3355.220876] LustreError: 123293:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3355.228032] LustreError: 123293:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3355.275969] Lustre: server umount lustre-MDT0000 complete [ 3358.356974] LustreError: 120843:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753745826 with bad export cookie 4358910879347688263 [ 3358.357452] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3358.365383] LustreError: 120843:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3358.538816] LustreError: 123494:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3358.541984] LustreError: 123494:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3358.737310] Lustre: server umount lustre-MDT0001 complete [ 3368.956906] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 3371.717434] Key type lgssc unregistered [ 3371.981167] LNet: 124021:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3373.032138] LNet: Removed LNI 192.168.202.105@tcp [ 3373.720104] Key type .llcrypt unregistered [ 3373.722487] Key type ._llcrypt unregistered [ 3394.370893] Key type ._llcrypt registered [ 3394.372745] Key type .llcrypt registered [ 3394.445649] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 3395.134035] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3395.163307] alg: No test for adler32 (adler32-zlib) [ 3396.183701] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 3396.368874] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 3396.375877] LNet: Accept secure, port 988 [ 3398.055268] Key type lgssc registered [ 3398.947941] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3405.729919] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3405.769893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3407.142156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3410.445722] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3417.576835] LustreError: 125408:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3417.590540] LustreError: 125408:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3417.662908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3418.034971] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3421.361492] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3426.240650] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3429.776639] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3434.624469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3434.895827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3438.932993] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3443.048644] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3452.340160] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:131 to 0x240000400:161) [ 3458.052799] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3461.344606] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3461.354510] Lustre: Skipped 2 previous similar messages [ 3464.723741] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3464.727306] Lustre: Skipped 2 previous similar messages [ 3466.808503] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3466.836103] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3470.387934] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.5@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3470.394307] Lustre: Skipped 2 previous similar messages [ 3472.700912] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.5@tcp 192.168.0.[2,10]@tcp [ 3476.590074] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.5/24@tcp in log lustre-MDT0000 [ 3476.600950] Lustre: Skipped 2 previous similar messages [ 3479.585764] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.5/24@tcp [ 3479.588475] Lustre: Skipped 1 previous similar message [ 3486.454406] LustreError: 128867:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3486.571416] Lustre: server umount lustre-OST0000 complete [ 3489.764705] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3489.770160] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3489.783496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3493.346370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3493.348343] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3498.464571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3498.466707] Lustre: Skipped 2 previous similar messages [ 3499.495280] LustreError: 129067:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3499.499940] LustreError: 129067:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3499.548173] Lustre: server umount lustre-MDT0000 complete [ 3502.332538] LustreError: 125387:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753745970 with bad export cookie 18426672519450741760 [ 3502.335562] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3502.342585] LustreError: 125387:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3511.790978] LustreError: 129269:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3511.796974] LustreError: 129269:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3511.925786] Lustre: server umount lustre-MDT0001 complete [ 3519.693511] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 3521.390607] Key type lgssc unregistered [ 3521.591485] LNet: 129798:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3522.661697] LNet: Removed LNI 192.168.202.105@tcp [ 3523.212279] Key type .llcrypt unregistered [ 3523.215269] Key type ._llcrypt unregistered [ 3542.971418] Key type ._llcrypt registered [ 3542.972607] Key type .llcrypt registered [ 3543.049239] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 3543.694833] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3543.787455] alg: No test for adler32 (adler32-zlib) [ 3544.798760] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 3544.973735] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 3544.977372] LNet: Accept secure, port 988 [ 3546.615176] Key type lgssc registered [ 3547.372811] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3552.615229] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3552.630277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3553.961236] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3553.982649] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.5/24@tcp [ 3553.998568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3556.919752] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3563.464765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3563.746517] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3563.760876] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.5/24@tcp [ 3563.801979] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3566.569960] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3570.903431] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3573.475712] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3578.049510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3578.281266] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3580.338031] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:169 to 0x240000400:193) [ 3581.909672] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3585.896152] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3598.820978] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3598.824663] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3598.831690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3599.842540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3599.843294] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3599.850478] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3602.671327] LustreError: 133588:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3602.803181] Lustre: server umount lustre-OST0000 complete [ 3610.081610] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3610.083358] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3610.084098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3610.084103] Lustre: Skipped 1 previous similar message [ 3610.095469] Lustre: Skipped 1 previous similar message [ 3615.203654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3615.207273] Lustre: Skipped 2 previous similar messages [ 3619.298885] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3619.372606] LustreError: 133789:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3619.376221] LustreError: 133789:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3619.423098] Lustre: server umount lustre-MDT0000 complete [ 3620.321161] LustreError: 131307:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3620.328819] LustreError: 131307:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 3621.993423] LustreError: 131287:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753746090 with bad export cookie 14700050649763009632 [ 3622.000894] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3622.087869] LustreError: 133992:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3622.090647] LustreError: 133992:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3622.260924] Lustre: server umount lustre-MDT0001 complete [ 3629.528783] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 3631.586784] Key type lgssc unregistered [ 3631.824702] LNet: 134519:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3632.870219] LNet: Removed LNI 192.168.202.105@tcp [ 3633.527917] Key type .llcrypt unregistered [ 3633.534369] Key type ._llcrypt unregistered [ 3652.415231] Key type ._llcrypt registered [ 3652.417262] Key type .llcrypt registered [ 3652.493460] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [ 3662.537947] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 3663.218671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3663.294527] alg: No test for adler32 (adler32-zlib) [ 3664.340788] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 3664.594455] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 3664.601742] LNet: Accept secure, port 988 [ 3666.248655] Key type lgssc registered [ 3667.064561] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3673.455158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3678.289613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3682.768216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3687.431332] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3691.676091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3691.706195] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3691.716892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3692.876040] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3692.904509] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3692.952142] Lustre: lustre-MDT0000: new disk, initializing [ 3693.037974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3693.048977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3695.667416] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3703.559244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3703.618034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3703.688294] Lustre: 137853:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3703.731362] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3703.734966] Lustre: Skipped 1 previous similar message [ 3703.784421] Lustre: lustre-MDT0001: new disk, initializing [ 3703.856343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3703.887142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3703.893558] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3706.698884] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3712.378218] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3714.945786] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3719.610978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3719.664813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3719.830342] Lustre: lustre-OST0000: new disk, initializing [ 3719.836345] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3719.887834] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3723.156062] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3726.361328] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3726.366140] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3726.447451] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3727.900387] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3730.154217] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3730.272157] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3732.948570] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3733.104335] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3736.547518] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3736.550773] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3736.553623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3736.562206] Lustre: Skipped 1 previous similar message [ 3736.573138] Lustre: Skipped 1 previous similar message [ 3740.462025] LustreError: 140760:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3740.587056] Lustre: server umount lustre-OST0000 complete [ 3744.736939] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3744.742259] LustreError: Skipped 1 previous similar message [ 3744.744940] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3744.752245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3746.785792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3746.786100] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3748.965641] LustreError: 140962:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3748.971309] LustreError: 140962:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3749.041150] Lustre: server umount lustre-MDT0000 complete [ 3751.592588] LustreError: 139595:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753746219 with bad export cookie 10999681603301293025 [ 3751.593098] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3751.599719] LustreError: 139595:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3751.705761] LustreError: 141163:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3751.708551] LustreError: 141163:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3751.853338] Lustre: server umount lustre-MDT0001 complete [ 3756.959799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3757.266610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3759.674597] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3762.657459] LustreError: 141639:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3762.665634] LustreError: 141639:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3764.950694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3765.190088] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3767.500601] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3771.211916] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3773.906135] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3778.474089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3778.682783] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3782.349456] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3786.088593] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3788.790459] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3788.911549] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3791.509824] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3791.621036] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3798.042745] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3798.083870] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3798.209878] Lustre: lustre-OST0001: new disk, initializing [ 3798.226665] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3798.302582] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3802.410372] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3806.293995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3806.298474] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3806.389545] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3807.863183] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3808.020193] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3810.357728] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3810.501624] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3812.692176] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff90ce43f1f800.ost_server_uuid 50 [ 3813.656522] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff90ce43f1f800.ost_server_uuid in FULL state after 0 sec [ 3822.056756] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3822.059609] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3822.070846] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3822.074283] Lustre: Skipped 2 previous similar messages [ 3823.992254] LustreError: 145838:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3824.001527] LustreError: 145838:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3824.047429] Lustre: server umount lustre-OST0001 complete [ 3826.145424] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3826.149230] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3826.155185] Lustre: Skipped 1 previous similar message [ 3826.160684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3827.173347] LustreError: 143227:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3827.180612] LustreError: 143227:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 3832.180126] LustreError: 146041:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3832.183569] LustreError: 146041:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3832.280683] Lustre: server umount lustre-OST0000 complete [ 3836.898316] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3836.907553] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3836.920482] Lustre: Skipped 1 previous similar message [ 3836.925469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3836.928111] Lustre: Skipped 2 previous similar messages [ 3840.461326] LustreError: 146242:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3840.463860] LustreError: 146242:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3840.516572] Lustre: server umount lustre-MDT0000 complete [ 3842.528676] LustreError: 141634:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3842.533924] LustreError: 141634:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 3842.732324] LustreError: 141620:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753746311 with bad export cookie 10999681603301293907 [ 3842.735659] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3842.737160] LustreError: 141620:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3842.954609] Lustre: server umount lustre-MDT0001 complete [ 3854.581961] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 3856.320257] Key type lgssc unregistered [ 3856.523745] LNet: 147427:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3857.573383] LNet: Removed LNI 192.168.202.105@tcp [ 3857.999362] Key type .llcrypt unregistered [ 3858.001204] Key type ._llcrypt unregistered [ 3877.201327] Key type ._llcrypt registered [ 3877.202827] Key type .llcrypt registered [ 3877.286930] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [ 3886.693289] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 3887.368897] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3887.376599] alg: No test for adler32 (adler32-zlib) [ 3888.358514] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 3888.574703] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 3888.578487] LNet: Accept secure, port 988 [ 3890.207176] Key type lgssc registered [ 3890.860761] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3896.492471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3900.460771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3904.399198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3908.673872] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3912.776270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3912.802716] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3912.814170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3913.940976] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3913.964524] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3914.012859] Lustre: lustre-MDT0000: new disk, initializing [ 3914.055793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3914.067428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3916.134303] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3922.958641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3922.998972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3923.049266] Lustre: 151334:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3923.083864] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3923.088208] Lustre: Skipped 1 previous similar message [ 3923.130991] Lustre: lustre-MDT0001: new disk, initializing [ 3923.183838] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3923.216800] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3923.222997] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3925.405470] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3929.955432] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3931.929173] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3936.508854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3936.560336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3936.756244] Lustre: lustre-OST0000: new disk, initializing [ 3936.759465] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3936.829060] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3940.201825] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3944.389023] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3946.681802] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3947.034806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3947.038828] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3947.105518] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3947.892755] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3950.476606] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3950.610474] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3952.101428] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3952.103346] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3952.106634] LustreError: Skipped 1 previous similar message [ 3952.115098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3952.117595] Lustre: Skipped 1 previous similar message [ 3953.914540] LustreError: 154247:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3954.012992] Lustre: server umount lustre-OST0000 complete [ 3957.218653] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3957.224237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3957.225579] Lustre: Skipped 2 previous similar messages [ 3962.244990] LustreError: 154448:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3962.247034] LustreError: 154448:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3962.299240] Lustre: server umount lustre-MDT0000 complete [ 3964.224153] LustreError: 151326:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753746432 with bad export cookie 15646573164243956500 [ 3964.226943] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3964.228719] LustreError: 151326:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3964.305241] LustreError: 154651:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3964.308256] LustreError: 154651:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3964.407357] Lustre: server umount lustre-MDT0001 complete [ 3969.252127] Lustre: DEBUG MARKER: == conf-sanity test 47a: server restart does not lose lru_resize ========================================================== 19:47:16 (1753746436) [ 3977.723246] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [ 3982.706335] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 3988.364324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3992.628648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3997.371488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4001.960737] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4006.863526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4006.922586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4007.106890] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4007.143828] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4007.227253] Lustre: lustre-MDT0000: new disk, initializing [ 4007.320586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4007.338992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4009.867494] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4017.375480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4017.426486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4017.497370] Lustre: 157621:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4017.544100] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4017.547496] Lustre: Skipped 1 previous similar message [ 4017.596579] Lustre: lustre-MDT0001: new disk, initializing [ 4017.696722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4017.724417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4017.729990] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4020.079558] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4024.548669] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4026.569358] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4030.221495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4030.264935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4030.419576] Lustre: lustre-OST0000: new disk, initializing [ 4030.426922] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4030.496193] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4034.077461] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4037.649309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4037.655568] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4037.719835] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4039.215678] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4042.178611] Lustre: Failing over lustre-OST0000 [ 4042.228211] LustreError: 160048:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4042.231249] LustreError: 160048:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4042.317159] Lustre: server umount lustre-OST0000 complete [ 4057.589198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4057.793158] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4057.812624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4057.824698] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4058.847102] Lustre: 149555:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1753746511/real 1753746511] req@00000000b6365c94 x1838936339049216/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1753746527 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4059.618172] Lustre: 149552:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1753746511/real 1753746511] req@000000005632f6c8 x1838936339049088/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1753746527 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4060.640296] Lustre: 149552:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1753746513/real 1753746513] req@00000000f9e59620 x1838936339049984/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1753746529 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4061.019178] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4061.766211] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4063.227932] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 4063.232849] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to (at 0@lo) [ 4064.287093] Lustre: 149552:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1753746516/real 1753746516] req@00000000f255369a x1838936339051008/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1753746532 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4064.642853] Lustre: Failing over lustre-MDT0000 [ 4064.750719] LustreError: 161026:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4064.753096] LustreError: 161026:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4064.825121] Lustre: server umount lustre-MDT0000 complete [ 4066.127621] LustreError: 157628:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4066.138174] LustreError: 157628:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 4068.321326] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4068.322778] LustreError: 157629:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4068.330298] Lustre: Skipped 2 previous similar messages [ 4068.341419] LustreError: 157629:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 4069.343156] Lustre: 149553:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1753746521/real 1753746521] req@000000007fa0bfb3 x1838936339052032/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1753746537 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4069.358684] Lustre: 149553:0:(client.c:2453:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4071.214472] LustreError: 157629:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4073.440511] LustreError: 157632:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4073.446735] LustreError: 157632:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 4078.562264] LustreError: 157632:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4078.574532] LustreError: 157632:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 4080.318727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4080.389521] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4080.574572] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4081.453801] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4082.477297] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4085.734921] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 4085.737022] Lustre: Skipped 1 previous similar message [ 4085.763450] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4085.790555] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4088.804578] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4088.808358] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4088.813181] Lustre: Skipped 1 previous similar message [ 4088.819240] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4088.831310] Lustre: Skipped 1 previous similar message [ 4090.851335] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4090.858162] Lustre: Skipped 1 previous similar message [ 4093.814401] LustreError: 161904:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4093.820275] LustreError: 161904:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4093.944781] Lustre: server umount lustre-OST0000 complete [ 4101.089255] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4101.091667] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4101.092747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4101.097808] Lustre: Skipped 2 previous similar messages [ 4102.496564] Lustre: server umount lustre-MDT0000 complete [ 4104.893165] LustreError: 157613:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753746573 with bad export cookie 15646573164243958607 [ 4104.895961] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4104.897895] LustreError: 157613:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4105.126268] Lustre: server umount lustre-MDT0001 complete [ 4112.988202] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 4115.169975] Key type lgssc unregistered [ 4115.386929] LNet: 162837:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4116.455071] LNet: Removed LNI 192.168.202.105@tcp [ 4116.973382] Key type .llcrypt unregistered [ 4116.976521] Key type ._llcrypt unregistered [ 4143.014329] Key type ._llcrypt registered [ 4143.015945] Key type .llcrypt registered [ 4143.068592] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 4143.526195] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4143.607336] alg: No test for adler32 (adler32-zlib) [ 4144.535849] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 4144.679663] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 4144.683709] LNet: Accept secure, port 988 [ 4146.319660] Key type lgssc registered [ 4147.004793] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4151.766527] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4151.780769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4153.170536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4155.527601] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4160.376702] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4163.172993] LustreError: 164354:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4163.180315] LustreError: 164354:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4163.186757] LustreError: 164354:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -5 [ 4163.266898] LustreError: 164870:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4163.384036] Lustre: server umount lustre-MDT0000 complete [ 4171.551197] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [ 4176.723300] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 4182.420575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4186.001068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4188.982559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4192.480693] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4196.480305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4196.536718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4196.714928] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4196.742478] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4196.792928] Lustre: lustre-MDT0000: new disk, initializing [ 4196.861218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4196.874257] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4199.273262] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4205.433373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4205.475094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4205.533889] Lustre: 167493:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4205.577983] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4205.583949] Lustre: Skipped 1 previous similar message [ 4205.629185] Lustre: lustre-MDT0001: new disk, initializing [ 4205.683181] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4205.706131] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4205.710470] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4207.728902] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4212.216228] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4214.260870] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4217.748766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4217.788433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4217.915113] Lustre: lustre-OST0000: new disk, initializing [ 4217.917549] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4217.954985] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4219.335432] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4219.339686] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4219.389895] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4220.447529] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4224.763369] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4227.347608] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4228.495308] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 19:51:35 (1753746695) [ 4393.441574] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4393.441743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4393.449485] Lustre: Skipped 1 previous similar message [ 4393.453368] Lustre: Skipped 1 previous similar message [ 4398.559760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4398.566477] Lustre: Skipped 3 previous similar messages [ 4403.679672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4403.685108] Lustre: Skipped 2 previous similar messages [ 4406.239157] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4406.291803] LustreError: 170987:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4406.295861] LustreError: 170987:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4406.385429] Lustre: server umount lustre-MDT0000 complete [ 4408.579071] LustreError: 168268:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753746876 with bad export cookie 9567977210960258536 [ 4408.580747] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4408.586132] LustreError: 168268:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4408.800023] LustreError: 167500:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4408.801724] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4408.807542] LustreError: 167500:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 4408.824785] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4413.921263] LustreError: 167500:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4413.921747] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4414.729367] LustreError: 171194:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4414.732309] LustreError: 171194:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4414.845596] Lustre: server umount lustre-MDT0001 complete [ 4431.327133] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4431.344659] LustreError: 171396:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4431.347818] LustreError: 171396:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4431.491322] Lustre: server umount lustre-OST0000 complete [ 4436.325214] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [ 4441.541996] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 4447.754677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4451.602057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4455.235755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4459.106551] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4463.047992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4463.097390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4463.237685] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4463.255556] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4463.301681] Lustre: lustre-MDT0000: new disk, initializing [ 4463.343897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4463.360328] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4465.542453] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4471.421645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4471.454263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4471.499555] Lustre: 173715:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4471.533578] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4471.536122] Lustre: Skipped 1 previous similar message [ 4471.578053] Lustre: lustre-MDT0001: new disk, initializing [ 4471.636116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4471.658911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4471.665482] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4473.454757] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4477.591368] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4479.504619] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4482.750505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4482.784576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4482.940451] Lustre: lustre-OST0000: new disk, initializing [ 4482.943372] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4485.593714] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4490.213593] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4492.315057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4492.323327] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4492.399886] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4492.441425] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4492.574857] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4494.696725] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4494.794075] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4497.380196] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4497.380519] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4497.381539] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4497.386158] Lustre: Skipped 1 previous similar message [ 4501.804099] LustreError: 176621:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4501.814240] LustreError: 176621:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4501.907634] Lustre: server umount lustre-OST0000 complete [ 4507.616687] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4507.617761] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4507.626297] Lustre: Skipped 1 previous similar message [ 4516.833018] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4516.835879] Lustre: Skipped 6 previous similar messages [ 4518.367134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4518.420033] LustreError: 176822:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4518.422415] LustreError: 176822:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4518.463255] Lustre: server umount lustre-MDT0000 complete [ 4520.102414] LustreError: 173752:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753746988 with bad export cookie 9567977210960417366 [ 4520.104923] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4520.110232] LustreError: 173752:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4520.290305] Lustre: server umount lustre-MDT0001 complete [ 4524.463417] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 19:56:31 (1753746991) [ 4532.252867] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [ 4537.014540] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 4541.827205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4544.894602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4548.103862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4551.673273] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4554.917890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4554.957390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4555.092692] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4555.111795] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4555.167823] Lustre: lustre-MDT0000: new disk, initializing [ 4555.211857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4555.214906] Lustre: Skipped 1 previous similar message [ 4555.222666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4557.162940] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4562.706068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4562.743447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4562.810483] Lustre: 179997:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4563.008229] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4564.963459] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4568.932814] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4570.671441] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4573.965391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4574.000058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4577.007326] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4578.829240] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4578.883169] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4581.384414] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4589.025732] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4589.029152] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4589.029743] LustreError: Skipped 1 previous similar message [ 4589.036062] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4589.041143] Lustre: Skipped 1 previous similar message [ 4590.707338] LustreError: 182527:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4590.709354] LustreError: 182527:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 4590.791620] Lustre: server umount lustre-OST0000 complete [ 4593.632569] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4598.763168] Lustre: server umount lustre-MDT0000 complete [ 4599.268295] LustreError: 180009:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4599.275441] LustreError: 180009:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 4600.569800] LustreError: 179990:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753747068 with bad export cookie 9567977210960418248 [ 4600.571130] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4600.575777] LustreError: 179990:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4600.772801] Lustre: server umount lustre-MDT0001 complete [ 4604.794840] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 19:57:52 (1753747072) [ 4612.705609] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [ 4617.220034] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 4621.372413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4624.330431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4627.130056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4629.874842] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4632.706433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4632.746068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4632.854694] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4632.872893] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4632.875062] Lustre: Skipped 3 previous similar messages [ 4632.917576] Lustre: lustre-MDT0000: new disk, initializing [ 4632.919836] Lustre: Skipped 2 previous similar messages [ 4632.959989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4632.964370] Lustre: Skipped 2 previous similar messages [ 4632.972802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4632.978251] Lustre: Skipped 2 previous similar messages [ 4634.601651] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4639.979570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4640.013204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4640.078174] Lustre: 185900:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4640.244332] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4642.000488] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4646.052242] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4647.804486] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4650.895198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4650.929089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4653.002303] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4653.522398] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4657.343758] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4661.938587] LustreError: 188424:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4661.942701] LustreError: 188424:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4662.025913] Lustre: server umount lustre-OST0000 complete [ 4665.826421] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4665.829979] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4665.837790] Lustre: Skipped 3 previous similar messages [ 4665.842638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4665.847480] Lustre: Skipped 3 previous similar messages [ 4674.563118] Lustre: server umount lustre-MDT0000 complete [ 4676.393332] LustreError: 185892:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753747144 with bad export cookie 9567977210960419333 [ 4676.397272] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4676.398416] LustreError: 185892:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4685.919266] Lustre: server umount lustre-MDT0001 complete [ 4691.532878] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 4692.849243] Key type lgssc unregistered [ 4693.025726] LNet: 189353:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4694.054814] LNet: Removed LNI 192.168.202.105@tcp [ 4694.423828] Key type .llcrypt unregistered [ 4694.425613] Key type ._llcrypt unregistered [ 4706.882993] Key type ._llcrypt registered [ 4706.884583] Key type .llcrypt registered [ 4706.943607] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 4707.503866] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4707.515865] alg: No test for adler32 (adler32-zlib) [ 4708.402740] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 4708.515083] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 4708.517823] LNet: Accept secure, port 988 [ 4710.119295] Key type lgssc registered [ 4710.669165] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4714.798576] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4714.811700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4716.070586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4718.051091] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4722.119820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4722.300600] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4724.135403] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4726.867465] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4728.849528] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4732.286704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4732.466696] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4735.240407] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4737.935279] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4740.648615] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4759.009034] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4759.012546] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4759.018993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4761.058838] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4761.059096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4761.059355] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4761.074853] Lustre: Skipped 1 previous similar message [ 4762.286296] LustreError: 192968:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4762.385573] Lustre: server umount lustre-OST0000 complete [ 4766.176830] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4766.180228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4766.183824] Lustre: Skipped 1 previous similar message [ 4770.508460] LustreError: 193169:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4770.511890] LustreError: 193169:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4770.557146] Lustre: server umount lustre-MDT0000 complete [ 4771.296133] LustreError: 190718:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4771.304308] LustreError: 190718:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 4772.491952] LustreError: 190700:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753747240 with bad export cookie 15566373669803536512 [ 4772.493096] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4772.497558] LustreError: 190700:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4772.572636] LustreError: 193371:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4772.575825] LustreError: 193371:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4772.710452] Lustre: server umount lustre-MDT0001 complete [ 4778.821532] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 4780.255338] Key type lgssc unregistered [ 4780.423614] LNet: 193897:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4781.477340] LNet: Removed LNI 192.168.202.105@tcp [ 4781.895908] Key type .llcrypt unregistered [ 4781.897430] Key type ._llcrypt unregistered [ 4794.092719] Key type ._llcrypt registered [ 4794.094317] Key type .llcrypt registered [ 4794.145532] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 4794.527893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4794.557017] alg: No test for adler32 (adler32-zlib) [ 4795.446522] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 4795.561314] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 4795.564116] LNet: Accept secure, port 988 [ 4797.175311] Key type lgssc registered [ 4797.708825] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4801.645094] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4801.663410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4802.921178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4804.589362] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4808.754738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4808.957825] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4810.723521] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4813.749704] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4815.923411] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4819.211450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4819.382268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4822.126622] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4824.616809] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4828.590929] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4838.882215] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4838.882578] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4838.887384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4838.887620] Lustre: Skipped 1 previous similar message [ 4838.893923] LustreError: Skipped 1 previous similar message [ 4841.293544] Lustre: lustre-OST0000: Not available for connect from 192.168.202.5@tcp (stopping) [ 4841.297037] Lustre: Skipped 1 previous similar message [ 4843.445167] LustreError: 197519:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4843.539281] Lustre: server umount lustre-OST0000 complete [ 4846.551160] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff90ce44878000.ost_server_uuid 50 [ 4847.621753] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 4859.361118] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4859.369794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4859.372328] Lustre: Skipped 1 previous similar message [ 4864.897157] LustreError: 197887:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4864.899613] LustreError: 197887:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4864.943864] Lustre: server umount lustre-MDT0000 complete [ 4865.504506] LustreError: 197780:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4865.511183] LustreError: 197780:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 4866.665660] LustreError: 196878:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753747335 with bad export cookie 14979812739589500955 [ 4866.666796] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4866.670178] LustreError: 196878:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4866.764480] LustreError: 198088:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4866.767477] LustreError: 198088:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4866.898952] Lustre: server umount lustre-MDT0001 complete [ 4871.288707] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 20:02:18 (1753747338) [ 4874.907889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4875.156166] LustreError: 198762:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4875.198555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4877.011554] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4880.353202] LustreError: 198762:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4881.049538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4881.214924] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4883.079612] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4885.954766] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4888.089700] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4891.483143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4891.655639] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4894.539351] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4897.284954] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4899.879465] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4900.593892] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4900.625579] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4900.712984] Lustre: lustre-OST0001: new disk, initializing [ 4900.715848] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4903.480172] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4907.817196] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4910.047984] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4910.051854] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4910.056877] Lustre: Skipped 1 previous similar message [ 4910.059582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4910.062658] Lustre: Skipped 2 previous similar messages [ 4910.597365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4910.602236] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4910.636858] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4915.683810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4915.685964] Lustre: Skipped 3 previous similar messages [ 4923.359144] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4923.381189] LustreError: 201930:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4923.386672] LustreError: 201930:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4923.411304] Lustre: server umount lustre-OST0000 complete [ 4923.695776] LustreError: 200350:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4925.929626] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4926.028939] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4927.822666] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4927.961398] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4928.817278] LustreError: 201937:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4928.829670] LustreError: 201937:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 4941.282810] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4941.282889] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4941.284728] LustreError: 201937:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4941.284737] LustreError: 201937:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 5 previous similar messages [ 4941.290316] Lustre: Skipped 1 previous similar message [ 4941.294555] LustreError: Skipped 1 previous similar message [ 4941.295599] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4941.314542] Lustre: Skipped 4 previous similar messages [ 4944.685236] LustreError: 202632:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 4944.688050] LustreError: 202632:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4944.763976] Lustre: server umount lustre-OST0001 complete [ 4946.401329] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4946.407427] Lustre: Skipped 1 previous similar message [ 4952.500295] Lustre: server umount lustre-MDT0000 complete [ 4954.445468] LustreError: 198743:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753747422 with bad export cookie 14979812739589502012 [ 4954.448175] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4954.450415] LustreError: 198743:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4954.533323] LustreError: 203035:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 4954.536470] LustreError: 203035:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 4954.654296] Lustre: server umount lustre-MDT0001 complete [ 4957.677906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4959.561016] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4961.286634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4963.207392] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4966.652249] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 20:03:54 (1753747434) [ 4969.966907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4970.105956] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4970.129426] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4970.232868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4970.236015] Lustre: Skipped 1 previous similar message [ 4972.047371] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4976.236437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4976.291622] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4976.314172] Lustre: 204526:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4978.173116] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4980.955686] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4982.706484] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4985.913689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4986.013220] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4988.864511] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4991.529208] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4991.586140] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4994.895522] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4994.940949] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4997.594620] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5000.044210] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5001.696190] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5001.696208] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5001.696701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5001.699447] LustreError: Skipped 1 previous similar message [ 5001.710543] Lustre: Skipped 5 previous similar messages [ 5007.531096] LustreError: 207695:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5007.533060] LustreError: 207695:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 5007.550921] Lustre: server umount lustre-OST0000 complete [ 5009.383038] LustreError: 206142:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5009.390233] LustreError: 206142:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 5019.618379] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5019.621396] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5019.626574] Lustre: Skipped 1 previous similar message [ 5024.678088] Lustre: server umount lustre-OST0001 complete [ 5027.807965] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5027.811354] LustreError: Skipped 1 previous similar message [ 5032.292141] Lustre: server umount lustre-MDT0000 complete [ 5033.844364] LustreError: 204517:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753747502 with bad export cookie 14979812739589503377 [ 5033.848166] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5033.849433] LustreError: 204517:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5034.013718] Lustre: server umount lustre-MDT0001 complete [ 5036.336319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5037.852472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5039.353619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5041.034717] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5043.999528] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 20:05:11 (1753747511) [ 5050.412445] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [ 5054.216269] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 5058.375431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5061.259955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5063.878402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5066.755488] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5069.886119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5069.924878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5070.045499] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5070.063513] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5070.106709] Lustre: lustre-MDT0000: new disk, initializing [ 5070.154687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5070.157874] Lustre: Skipped 3 previous similar messages [ 5070.166083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5071.888162] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5077.254586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5077.288393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5077.337684] Lustre: 212109:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5077.370873] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5077.373380] Lustre: Skipped 1 previous similar message [ 5077.413386] Lustre: lustre-MDT0001: new disk, initializing [ 5077.470912] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5077.475218] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5079.191208] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5083.073702] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5084.836967] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5087.810079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5087.848985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5087.964488] Lustre: lustre-OST0000: new disk, initializing [ 5087.967209] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5090.490279] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5094.590095] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5096.308572] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5097.994426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5097.998680] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5098.052508] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5098.466406] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 5100.122312] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5100.215446] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5103.074715] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5103.075900] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5103.078475] LustreError: Skipped 1 previous similar message [ 5103.083849] Lustre: Skipped 3 previous similar messages [ 5103.084715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5103.090615] Lustre: Skipped 9 previous similar messages [ 5107.118104] LustreError: 215027:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5107.120918] LustreError: 215027:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 5107.194490] Lustre: server umount lustre-OST0000 complete [ 5114.808841] Lustre: server umount lustre-MDT0000 complete [ 5116.483171] LustreError: 212099:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753747584 with bad export cookie 14979812739589504924 [ 5116.486169] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5116.488594] LustreError: 212099:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5116.657531] Lustre: server umount lustre-MDT0001 complete [ 5120.439596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5120.619078] LustreError: 215906:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5120.627481] LustreError: 215906:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 6 previous similar messages [ 5122.192478] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5125.875953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5127.681705] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5129.990682] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5131.544762] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5134.317538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5134.447499] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5134.450397] Lustre: Skipped 4 previous similar messages [ 5136.554713] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5138.513173] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5140.025056] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5145.231079] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec [ 5146.555543] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5146.635938] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5150.177592] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5150.180980] LustreError: Skipped 1 previous similar message [ 5153.451248] Lustre: server umount lustre-OST0000 complete [ 5155.621534] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5157.027838] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5180.986940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5183.180293] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5185.221619] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5185.927895] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5189.162979] LustreError: 220264:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5189.165650] LustreError: 220264:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 5189.231078] Lustre: server umount lustre-OST0000 complete [ 5190.690081] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5190.695426] Lustre: Skipped 5 previous similar messages [ 5190.698635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5190.701552] Lustre: Skipped 5 previous similar messages [ 5192.785554] Lustre: server umount lustre-MDT0000 complete [ 5194.270532] LustreError: 215888:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753747662 with bad export cookie 14979812739589505792 [ 5194.273133] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5194.275928] LustreError: 215888:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5194.431109] Lustre: server umount lustre-MDT0001 complete [ 5197.752263] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 20:07:45 (1753747665) [ 5200.098379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5200.249618] LustreError: 221342:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5201.645404] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5204.906577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5206.478323] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5208.596147] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5210.161779] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5212.697337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5214.861489] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5216.792255] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5218.246914] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5226.533928] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 5227.977657] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5228.067512] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5230.523201] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5230.548985] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5230.615985] Lustre: lustre-OST0001: new disk, initializing [ 5230.618615] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5232.683454] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5236.095457] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5237.556699] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5238.275064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5238.279153] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5238.308449] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5238.668926] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 5240.123700] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5240.210521] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5243.361879] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5243.364505] LustreError: Skipped 1 previous similar message [ 5246.980518] Lustre: server umount lustre-OST0001 complete [ 5249.469267] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5249.563816] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5251.198843] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5251.310187] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5275.480290] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5275.559938] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5275.563523] Lustre: Skipped 5 previous similar messages [ 5277.768276] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5280.032928] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5280.741078] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5280.745886] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to (at 0@lo) [ 5280.745987] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5280.820403] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5283.706785] Lustre: server umount lustre-OST0001 complete [ 5302.153804] Lustre: server umount lustre-OST0000 complete [ 5310.120124] Lustre: server umount lustre-MDT0000 complete [ 5311.875083] LustreError: 221323:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753747780 with bad export cookie 14979812739589506723 [ 5311.877446] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5311.879472] LustreError: 221323:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5326.303188] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5326.341758] LustreError: 227816:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 5326.344856] LustreError: 227816:0:(obd_class.h:479:obd_check_dev()) Skipped 37 previous similar messages [ 5326.442680] Lustre: server umount lustre-MDT0001 complete [ 5329.003266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5330.772031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5332.647371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5334.479567] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5337.669068] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 20:10:05 (1753747805) [ 5340.529227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5340.647055] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5340.660368] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5342.351274] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5346.040051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5346.090806] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5346.104819] Lustre: 229310:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5347.841309] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5350.128158] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5351.719977] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5354.688617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5354.773723] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5357.107865] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5359.342313] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5369.797386] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5369.837114] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5372.464798] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5374.725383] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5378.565577] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 5380.263608] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5380.372060] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5381.965536] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5382.050672] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5383.543923] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff90ce48050000.ost_server_uuid 50 [ 5384.187477] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff90ce48050000.ost_server_uuid in FULL state after 0 sec [ 5384.580347] Lustre: Permanently deactivating lustre-OST0001 [ 5384.586562] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 5385.215346] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 5385.620596] Lustre: Permanently reactivating lustre-OST0001 [ 5392.971177] Lustre: server umount lustre-OST0001 complete [ 5394.418554] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5394.423781] Lustre: Skipped 9 previous similar messages [ 5394.426757] LustreError: 230923:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5394.433717] LustreError: 230923:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 27 previous similar messages [ 5398.496895] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5398.500993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5398.501244] LustreError: Skipped 3 previous similar messages [ 5398.504301] Lustre: Skipped 11 previous similar messages [ 5400.832989] Lustre: server umount lustre-OST0000 complete [ 5411.857228] Lustre: server umount lustre-MDT0000 complete [ 5413.753174] LustreError: 229300:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753747882 with bad export cookie 14979812739589507997 [ 5413.759222] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5414.879191] LustreError: 233438:0:(osp_dev.c:694:osp_process_config()) lustre-OST0001-osc-MDT0001: unknown param osc.active=1 [ 5414.887437] LustreError: 229312:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 5415.080165] Lustre: server umount lustre-MDT0001 complete [ 5417.697986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5419.251787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5420.719579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5422.371062] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5425.106554] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 20:11:32 (1753747892) [ 5425.516308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5428.548505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5428.706693] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5428.726460] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5428.731549] Lustre: Skipped 5 previous similar messages [ 5430.716844] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5434.359547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5434.401456] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5434.415819] Lustre: 235418:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5436.071059] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5438.452380] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5439.978752] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5442.728739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5442.755030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5442.847705] Lustre: Permanently deactivating lustre-OST0000 [ 5442.854708] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5443.766790] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5443.770097] Lustre: Skipped 1 previous similar message [ 5443.772067] LustreError: 237186:0:(osp_precreate.c:654:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 5443.777094] LustreError: 237186:0:(osp_precreate.c:1358:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 5445.088218] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5447.115545] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5449.692383] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5451.981236] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5452.574826] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 5454.216855] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5454.933473] Lustre: Permanently reactivating lustre-OST0000 [ 5462.521322] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 5462.524700] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5462.528404] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.202.105@tcp (at 0@lo) [ 5462.531559] Lustre: Skipped 1 previous similar message [ 5462.541798] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 5462.549180] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5469.316707] Lustre: server umount lustre-OST0001 complete [ 5476.781356] Lustre: server umount lustre-OST0000 complete [ 5484.512894] Lustre: server umount lustre-MDT0000 complete [ 5485.910107] LustreError: 236140:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753747954 with bad export cookie 14979812739589509705 [ 5485.913692] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5485.914514] LustreError: 236140:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5486.074795] Lustre: server umount lustre-MDT0001 complete [ 5491.284087] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 5492.456090] Key type lgssc unregistered [ 5492.600485] LNet: 239780:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5493.669403] LNet: Removed LNI 192.168.202.105@tcp [ 5493.999195] Key type .llcrypt unregistered [ 5494.000743] Key type ._llcrypt unregistered [ 5504.690023] Key type ._llcrypt registered [ 5504.691082] Key type .llcrypt registered [ 5504.731941] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 5505.087600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5505.136947] alg: No test for adler32 (adler32-zlib) [ 5506.010053] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 5506.097789] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 5506.099941] LNet: Accept secure, port 988 [ 5507.695183] Key type lgssc registered [ 5508.149779] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5509.562599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5512.367531] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5512.378385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5513.599863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5515.277273] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5519.240951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5519.271811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5519.319198] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5519.332501] Lustre: 241183:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5519.338541] Lustre: Permanently deactivating lustre-MDT0001 [ 5519.344084] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5519.490311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5521.135495] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5521.138572] LustreError: 241226:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5521.144176] LustreError: 241226:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5521.150468] LustreError: 241226:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 1, failed: rc = -108 [ 5521.169392] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5523.612845] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5525.323146] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5528.337464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5528.518432] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5529.574284] LustreError: 242802:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5529.584883] LustreError: 242802:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 5531.120726] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5533.329701] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5534.690250] LustreError: 242802:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5536.244866] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5536.370918] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5538.416884] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 5538.417964] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 5538.418462] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 5538.906519] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5541.294728] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5543.088628] Lustre: Permanently reactivating lustre-MDT0001 [ 5543.097129] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 5543.100203] Lustre: Skipped 2 previous similar messages [ 5548.350097] Lustre: lustre-MDT0001: Client 1e9efdd3-3069-4eff-8017-b22a0724b1bc (at 192.168.202.5@tcp) reconnecting [ 5550.853769] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5551.657952] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5553.839748] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 5553.936865] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 5555.574392] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 5555.668774] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 5556.703473] Lustre: Permanently deactivating lustre-MDT0001 [ 5556.712569] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5556.715349] Lustre: Skipped 2 previous similar messages [ 5566.468104] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5570.529085] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5570.530242] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5570.531227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5570.538352] Lustre: Skipped 3 previous similar messages [ 5575.649984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5575.652263] Lustre: Skipped 6 previous similar messages [ 5576.256153] LustreError: 245323:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 5576.283363] Lustre: server umount lustre-MDT0000 complete [ 5577.643269] LustreError: 245072:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753748046 with bad export cookie 13429055525712659044 [ 5577.646484] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5577.648058] LustreError: 245072:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5577.711428] LustreError: 245525:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 5577.714447] LustreError: 245525:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5577.791557] Lustre: server umount lustre-MDT0001 complete [ 5585.260774] LustreError: 245725:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 5585.263838] LustreError: 245725:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5585.280149] Lustre: server umount lustre-OST0000 complete [ 5589.578741] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 20:14:17 (1753748057) [ 5593.334223] LustreError: 246729:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5593.336191] LustreError: 246729:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5593.406145] Lustre: server umount lustre-OST0001 complete [ 5595.628968] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [ 5599.182561] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 5603.067839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5605.437589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5607.794795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5610.101063] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5612.731693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5612.755551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5612.841902] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5612.845584] Lustre: Skipped 2 previous similar messages [ 5612.855520] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5612.895374] Lustre: lustre-MDT0000: new disk, initializing [ 5612.919648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5612.927856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5614.284300] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5618.610055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5618.637441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5618.671476] Lustre: 248894:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5618.696513] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5618.698883] Lustre: Skipped 1 previous similar message [ 5618.736813] Lustre: lustre-MDT0001: new disk, initializing [ 5618.787777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5618.791729] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5620.261061] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5623.428553] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5624.661528] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5627.005107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5627.026374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5627.113141] Lustre: lustre-OST0000: new disk, initializing [ 5627.114600] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5627.141740] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5627.143978] Lustre: Skipped 1 previous similar message [ 5628.967780] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5632.220379] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5636.603698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5636.607981] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5636.645371] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5637.945621] LustreError: 249722:0:(mdt_handler.c:3249:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5642.186542] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5642.212033] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5642.265706] Lustre: lustre-OST0001: new disk, initializing [ 5642.267400] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5644.164770] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5644.858382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5644.861256] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5644.889375] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5647.374675] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5647.930213] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5648.039078] LustreError: 249722:0:(mdt_handler.c:3249:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5648.045804] LustreError: 249722:0:(mdt_handler.c:3249:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5658.135081] LustreError: 249722:0:(mdt_handler.c:3249:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5660.130917] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5660.131311] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5660.133632] LustreError: Skipped 1 previous similar message [ 5660.134284] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5660.137948] Lustre: Skipped 1 previous similar message [ 5660.145173] Lustre: Skipped 1 previous similar message [ 5665.069380] LustreError: 252387:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5665.072264] LustreError: 252387:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5665.091443] Lustre: server umount lustre-OST0001 complete [ 5665.250666] LustreError: 251229:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5665.256547] LustreError: 251229:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 5670.371534] LustreError: 251229:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5670.377626] LustreError: 251229:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 5675.490279] LustreError: 250610:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5675.495957] LustreError: 250610:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 5677.536454] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5677.536683] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5677.538528] LustreError: Skipped 1 previous similar message [ 5677.541988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5677.544245] Lustre: Skipped 2 previous similar messages [ 5685.729522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5685.730215] LustreError: 250611:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5685.731616] Lustre: Skipped 3 previous similar messages [ 5685.737159] LustreError: 250611:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 5691.359130] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5691.380094] LustreError: 252610:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5691.381979] LustreError: 252610:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5691.440726] Lustre: server umount lustre-OST0000 complete [ 5695.456064] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5695.459403] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5695.464438] Lustre: Skipped 1 previous similar message [ 5695.467099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5695.469323] Lustre: Skipped 2 previous similar messages [ 5698.903836] Lustre: server umount lustre-MDT0000 complete [ 5700.115598] LustreError: 250627:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753748168 with bad export cookie 13429055525712661634 [ 5700.117958] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5700.120917] LustreError: 250627:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5714.399179] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5714.435244] LustreError: 253015:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 5714.436773] LustreError: 253015:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 5714.512598] Lustre: server umount lustre-MDT0001 complete [ 5719.173767] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 5720.337149] Key type lgssc unregistered [ 5720.479490] LNet: 253544:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5721.509549] LNet: Removed LNI 192.168.202.105@tcp [ 5721.845721] Key type .llcrypt unregistered [ 5721.847049] Key type ._llcrypt unregistered [ 5733.726134] Key type ._llcrypt registered [ 5733.727390] Key type .llcrypt registered [ 5733.773679] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [ 5739.908117] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 5740.362067] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5740.379019] alg: No test for adler32 (adler32-zlib) [ 5741.237368] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 5741.322349] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 5741.324156] LNet: Accept secure, port 988 [ 5742.911175] Key type lgssc registered [ 5743.259140] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5746.440680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5748.571911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5750.597663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5752.746793] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5754.959215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5754.979433] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5754.984944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5756.069683] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5756.083343] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5756.119682] Lustre: lustre-MDT0000: new disk, initializing [ 5756.143876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5756.152242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5757.448945] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5761.674349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5761.694274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5761.726641] Lustre: 257450:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5761.749042] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5761.750505] Lustre: Skipped 1 previous similar message [ 5761.782948] Lustre: lustre-MDT0001: new disk, initializing [ 5761.812594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5761.824173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5761.827683] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5763.126275] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5766.304218] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5767.581488] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5769.983207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5770.010729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5770.110702] Lustre: lustre-OST0000: new disk, initializing [ 5770.113393] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5770.144319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5772.113852] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5773.112637] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5773.120857] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5773.150992] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5775.287721] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5776.648517] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5776.745199] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5778.091206] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5778.165525] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5783.520469] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5783.521129] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5783.522674] LustreError: Skipped 1 previous similar message [ 5783.526330] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5783.529450] Lustre: Skipped 1 previous similar message [ 5784.877326] LustreError: 260357:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5784.940549] Lustre: server umount lustre-OST0000 complete [ 5787.615881] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5787.618600] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5787.622651] Lustre: Skipped 1 previous similar message [ 5787.624534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5788.640374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5788.640866] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5792.380536] LustreError: 260558:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5792.382952] LustreError: 260558:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5792.413598] Lustre: server umount lustre-MDT0000 complete [ 5793.694240] LustreError: 257439:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753748262 with bad export cookie 7360264398576921017 [ 5793.697620] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5793.698315] LustreError: 257439:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5793.756836] LustreError: 260760:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5793.758542] LustreError: 260760:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5793.841896] Lustre: server umount lustre-MDT0001 complete [ 5796.814415] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 20:17:44 (1753748264) [ 5799.100219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5799.260855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5800.599576] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5803.625567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5803.777860] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5805.223379] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5807.169557] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5808.409449] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5810.690525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5812.708058] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5814.536988] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5865.442310] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5865.442838] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5865.446038] LustreError: Skipped 1 previous similar message [ 5865.446781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5865.449413] Lustre: Skipped 1 previous similar message [ 5865.453577] Lustre: Skipped 2 previous similar messages [ 5868.972110] LustreError: 263751:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5868.975248] LustreError: 263751:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5869.052296] Lustre: server umount lustre-OST0000 complete [ 5870.657730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 5875.232331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5875.348730] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5875.350568] Lustre: Skipped 1 previous similar message [ 5877.436492] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5879.399571] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5885.606267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5885.610595] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5885.610640] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.105@tcp (at 0@lo) [ 5885.617019] Lustre: Skipped 1 previous similar message [ 5916.643374] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5916.643412] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5916.648587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5916.653778] Lustre: Skipped 1 previous similar message [ 5919.402074] LustreError: 265406:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5919.404745] LustreError: 265406:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5919.482512] Lustre: server umount lustre-OST0000 complete [ 5921.761598] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5921.764966] LustreError: Skipped 1 previous similar message [ 5926.880081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5926.883670] Lustre: Skipped 4 previous similar messages [ 5927.118918] Lustre: server umount lustre-MDT0000 complete [ 5928.400232] LustreError: 261410:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753748396 with bad export cookie 7360264398576921885 [ 5928.403173] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5928.404315] LustreError: 261410:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5928.467246] LustreError: 265810:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5928.468751] LustreError: 265810:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 5928.557265] Lustre: server umount lustre-MDT0001 complete [ 5933.438712] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 5934.502740] Key type lgssc unregistered [ 5934.657480] LNet: 266336:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5935.717311] LNet: Removed LNI 192.168.202.105@tcp [ 5936.023535] Key type .llcrypt unregistered [ 5936.024473] Key type ._llcrypt unregistered [ 5945.580989] Key type ._llcrypt registered [ 5945.582408] Key type .llcrypt registered [ 5945.618191] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 5945.904691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5945.911704] alg: No test for adler32 (adler32-zlib) [ 5946.773862] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 5946.858131] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 5946.860640] LNet: Accept secure, port 988 [ 5948.447214] Key type lgssc registered [ 5948.813106] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5951.701097] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5951.709820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5952.876891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5954.233137] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5957.097428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5957.208668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5958.442830] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5960.250844] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5961.563490] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5963.896825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5964.003137] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5965.772559] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5967.598755] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5978.342797] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 5992.928290] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5992.931039] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5992.935941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5993.953783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5993.954368] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5993.954517] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5993.955748] Lustre: Skipped 1 previous similar message [ 5997.165105] LustreError: 270720:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5997.224278] Lustre: server umount lustre-OST0000 complete [ 5999.072631] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5999.073062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5999.077100] Lustre: Skipped 1 previous similar message [ 6004.192093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6004.193832] Lustre: Skipped 2 previous similar messages [ 6004.608567] LustreError: 270922:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6004.610725] LustreError: 270922:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6004.643165] Lustre: server umount lustre-MDT0000 complete [ 6005.916152] LustreError: 268413:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753748474 with bad export cookie 13193196862633266335 [ 6005.918826] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6005.920685] LustreError: 268413:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6005.983397] LustreError: 271123:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6005.985252] LustreError: 271123:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6006.061783] Lustre: server umount lustre-MDT0001 complete [ 6010.720658] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 6011.746960] Key type lgssc unregistered [ 6011.877370] LNet: 271650:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6012.901336] LNet: Removed LNI 192.168.202.105@tcp [ 6013.203942] Key type .llcrypt unregistered [ 6013.205038] Key type ._llcrypt unregistered [ 6020.113050] Key type ._llcrypt registered [ 6020.114313] Key type .llcrypt registered [ 6020.156295] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 6020.465862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6020.472936] alg: No test for adler32 (adler32-zlib) [ 6021.331451] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 6021.408865] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 6021.411588] LNet: Accept secure, port 988 [ 6022.999123] Key type lgssc registered [ 6023.396623] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6026.169650] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6026.177712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6027.329420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6028.611109] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6031.402577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6031.538850] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6032.836909] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6034.649209] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6035.857129] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6038.033815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6038.140791] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6039.843210] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6041.552863] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6052.517326] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6062.047817] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6062.049911] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6062.055817] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6063.073403] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6063.074669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6063.077918] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6063.079358] Lustre: Skipped 1 previous similar message [ 6066.862103] LustreError: 275214:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6066.930112] Lustre: server umount lustre-OST0000 complete [ 6068.192191] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6068.192638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6068.195696] Lustre: Skipped 1 previous similar message [ 6073.312571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6073.316013] Lustre: Skipped 2 previous similar messages [ 6074.175768] LustreError: 275416:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6074.179170] LustreError: 275416:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6074.210243] Lustre: server umount lustre-MDT0000 complete [ 6075.535635] LustreError: 274426:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753748543 with bad export cookie 2132434606328738250 [ 6075.538948] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6075.541797] LustreError: 274426:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6075.608285] LustreError: 275618:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6075.610655] LustreError: 275618:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6075.694899] Lustre: server umount lustre-MDT0001 complete [ 6080.895129] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 6082.048493] Key type lgssc unregistered [ 6082.195387] LNet: 276144:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6083.237328] LNet: Removed LNI 192.168.202.105@tcp [ 6083.565110] Key type .llcrypt unregistered [ 6083.566385] Key type ._llcrypt unregistered [ 6093.229075] Key type ._llcrypt registered [ 6093.230413] Key type .llcrypt registered [ 6093.275459] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 6093.644768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6093.666064] alg: No test for adler32 (adler32-zlib) [ 6094.533569] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 6094.635117] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 6094.636907] LNet: Accept secure, port 988 [ 6096.223145] Key type lgssc registered [ 6096.628877] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6099.509203] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6099.516971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6100.685413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6102.154925] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6105.003974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6105.116576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6106.443685] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6108.340480] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6109.492864] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6111.889250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6112.024123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6113.908299] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6115.693314] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6126.377367] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6135.776550] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6135.779288] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6135.784064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6136.802158] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6136.802264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6136.802514] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6136.811227] Lustre: Skipped 1 previous similar message [ 6140.009614] LustreError: 280542:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6140.067648] Lustre: server umount lustre-OST0000 complete [ 6141.920484] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6141.921040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6141.925083] Lustre: Skipped 1 previous similar message [ 6147.039915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6147.042230] Lustre: Skipped 2 previous similar messages [ 6147.449078] LustreError: 280744:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6147.451595] LustreError: 280744:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6147.476826] Lustre: server umount lustre-MDT0000 complete [ 6148.693152] LustreError: 279128:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753748617 with bad export cookie 2064147870728553476 [ 6148.695913] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6148.697114] LustreError: 279128:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6148.752837] LustreError: 280946:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6148.755013] LustreError: 280946:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6148.828154] Lustre: server umount lustre-MDT0001 complete [ 6153.274559] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 6154.345641] Key type lgssc unregistered [ 6154.484452] LNet: 281472:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6155.557310] LNet: Removed LNI 192.168.202.105@tcp [ 6155.844473] Key type .llcrypt unregistered [ 6155.845806] Key type ._llcrypt unregistered [ 6162.651703] Key type ._llcrypt registered [ 6162.652649] Key type .llcrypt registered [ 6162.702599] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 6163.100298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6163.136269] alg: No test for adler32 (adler32-zlib) [ 6163.992819] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 6164.089241] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 6164.091410] LNet: Accept secure, port 988 [ 6165.671173] Key type lgssc registered [ 6166.066762] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6168.924498] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6168.933682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6170.142566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6171.593405] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6174.749530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6174.888174] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6176.230376] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6178.251977] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6179.529895] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6181.833760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6181.942334] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6183.779795] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6185.444802] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6195.235903] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 6200.292522] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6200.292945] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6200.293606] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6200.297229] Lustre: Skipped 1 previous similar message [ 6201.712084] LustreError: 285070:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6201.779628] Lustre: server umount lustre-OST0000 complete [ 6205.113721] LustreError: 285272:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6205.115515] LustreError: 285272:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6205.140040] Lustre: server umount lustre-MDT0000 complete [ 6205.408667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6205.409799] LustreError: 282648:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6205.412924] Lustre: Skipped 1 previous similar message [ 6205.421179] LustreError: 282648:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 6206.479353] LustreError: 284984:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753748674 with bad export cookie 12534012728152168185 [ 6206.482849] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6206.483443] LustreError: 284984:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6206.559175] LustreError: 285473:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6206.561388] LustreError: 285473:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6206.668874] Lustre: server umount lustre-MDT0001 complete [ 6211.531619] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 6212.786880] Key type lgssc unregistered [ 6212.917402] LNet: 285999:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6213.989364] LNet: Removed LNI 192.168.202.105@tcp [ 6214.301977] Key type .llcrypt unregistered [ 6214.302985] Key type ._llcrypt unregistered [ 6218.142235] Key type ._llcrypt registered [ 6218.143266] Key type .llcrypt registered [ 6218.186731] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 6224.146859] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [ 6230.298768] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 6230.698503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6230.704929] alg: No test for adler32 (adler32-zlib) [ 6231.558554] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 6231.642900] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 6231.645059] LNet: Accept secure, port 988 [ 6233.223154] Key type lgssc registered [ 6233.570480] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6236.903809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6239.309952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6241.460582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6243.512619] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6245.913709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6245.939901] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6245.947521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6247.037890] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6247.049817] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6247.088736] Lustre: lustre-MDT0000: new disk, initializing [ 6247.114600] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6247.123089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6248.591137] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6252.902218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6252.926502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6252.961397] Lustre: 289460:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6252.995683] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6252.998595] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6253.001405] Lustre: Skipped 1 previous similar message [ 6253.039780] Lustre: lustre-MDT0001: new disk, initializing [ 6253.079083] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6253.095639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6253.100590] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6254.498284] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6257.651330] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6258.849258] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6261.052817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6261.073879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6261.164292] Lustre: lustre-OST0000: new disk, initializing [ 6261.165957] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6261.168814] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 6261.202381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6262.994365] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6266.022132] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6266.347944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6266.352763] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6266.366474] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6267.327811] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6267.409070] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6268.760116] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6268.856137] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6271.457141] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6271.458532] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6271.460248] LustreError: Skipped 1 previous similar message [ 6271.463620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6271.466784] Lustre: Skipped 1 previous similar message [ 6275.628773] LustreError: 292372:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6275.697260] Lustre: server umount lustre-OST0000 complete [ 6278.624159] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6278.628109] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6278.634059] Lustre: Skipped 1 previous similar message [ 6278.636929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6281.697783] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6283.197418] LustreError: 292573:0:(obd_class.h:479:obd_check_dev()) Device 20 not setup [ 6283.200033] LustreError: 292573:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6283.232838] Lustre: server umount lustre-MDT0000 complete [ 6284.606362] LustreError: 290234:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753748752 with bad export cookie 10198637687672385443 [ 6284.609361] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6284.609435] LustreError: 290234:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6284.677173] LustreError: 292774:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6284.680270] LustreError: 292774:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6284.762296] Lustre: server umount lustre-MDT0001 complete [ 6287.701550] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 20:25:55 (1753748755) [ 6290.110027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6290.314957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6291.762946] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6294.758218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6294.899607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6296.359312] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6298.164772] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6299.437141] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6301.749204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6303.857393] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6305.701778] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6320.609351] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6320.611185] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6320.613172] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6320.617496] Lustre: Skipped 1 previous similar message [ 6320.619653] Lustre: Skipped 2 previous similar messages [ 6325.228145] LustreError: 295696:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6325.229718] LustreError: 295696:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6325.283947] Lustre: server umount lustre-OST0000 complete [ 6329.825353] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 6329.827156] LustreError: Skipped 1 previous similar message [ 6329.828087] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6329.831054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6329.832881] Lustre: Skipped 1 previous similar message [ 6332.674055] LustreError: 295897:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6332.676189] LustreError: 295897:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6332.709042] Lustre: server umount lustre-MDT0000 complete [ 6334.033107] LustreError: 293425:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753748802 with bad export cookie 10198637687672386325 [ 6334.036118] LustreError: 293425:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6334.036972] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6334.179325] Lustre: server umount lustre-MDT0001 complete [ 6339.069085] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 6340.180247] Key type lgssc unregistered [ 6340.315407] LNet: 296627:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6341.349280] LNet: Removed LNI 192.168.202.105@tcp [ 6341.650234] Key type .llcrypt unregistered [ 6341.651165] Key type ._llcrypt unregistered [ 6349.894325] Key type ._llcrypt registered [ 6349.896448] Key type .llcrypt registered [ 6349.941383] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [ 6355.568058] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 6355.820332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6355.839980] alg: No test for adler32 (adler32-zlib) [ 6356.684779] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 6356.753758] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 6356.755034] LNet: Accept secure, port 988 [ 6358.327208] Key type lgssc registered [ 6358.698140] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6361.818434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6363.932750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6365.952098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6368.001489] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6369.727056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6371.496951] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6373.903811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6373.925990] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6373.932479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6375.018962] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6375.029590] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6375.069483] Lustre: lustre-MDT0000: new disk, initializing [ 6375.095427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6375.102681] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6376.466875] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6380.677597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6380.702051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6380.735096] Lustre: 300384:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6380.760543] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6380.763071] Lustre: Skipped 1 previous similar message [ 6380.801264] Lustre: lustre-MDT0001: new disk, initializing [ 6380.833852] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6380.848247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6380.852299] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6382.192037] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6385.241276] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6386.486203] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6388.815181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6388.835086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6388.933725] Lustre: lustre-OST2710: new disk, initializing [ 6388.936166] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 6388.965889] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 6390.943526] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6394.115715] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 6395.900391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 6395.904097] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 6395.938660] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 6396.548551] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6396.572055] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6396.632229] Lustre: lustre-OST03e8: new disk, initializing [ 6396.634291] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 6396.671246] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 6398.715327] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6400.556116] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 6403.138549] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 6403.229405] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6404.448780] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 6404.522802] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6405.123927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 6405.127155] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 6405.151810] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 6405.796601] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 6405.870197] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6407.142651] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 6407.216231] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6410.210385] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6410.211637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6410.214041] Lustre: Skipped 3 previous similar messages [ 6410.218225] Lustre: Skipped 3 previous similar messages [ 6414.266411] LustreError: 304674:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6414.290327] Lustre: server umount lustre-MDT0000 complete [ 6415.434794] LustreError: 301161:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753748883 with bad export cookie 651605352202671860 [ 6415.438323] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6415.438952] LustreError: 301161:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6415.496788] LustreError: 304881:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6415.498524] LustreError: 304881:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6415.566072] Lustre: server umount lustre-MDT0001 complete [ 6427.639128] LustreError: 305081:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6427.641959] LustreError: 305081:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6427.658288] Lustre: server umount lustre-OST2710 complete [ 6439.216723] LustreError: 305283:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6439.220019] LustreError: 305283:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6439.289188] Lustre: server umount lustre-OST03e8 complete [ 6445.330736] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [ 6448.236096] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 6451.628502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6453.770150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6455.729591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6457.831058] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6460.099045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6460.121298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6460.200962] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6460.214292] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6460.249039] Lustre: lustre-MDT0000: new disk, initializing [ 6460.273036] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6460.280111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6461.574841] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6465.668704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6465.690172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6465.719521] Lustre: 308059:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6465.815900] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6467.068101] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6470.064357] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6471.202232] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6473.580256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6473.609519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6473.717655] Lustre: lustre-OST0000: new disk, initializing [ 6473.719507] Lustre: Skipped 1 previous similar message [ 6473.722092] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6473.724820] Lustre: Skipped 2 previous similar messages [ 6473.767548] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6473.770701] Lustre: Skipped 1 previous similar message [ 6475.725278] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6477.111033] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6477.114091] Lustre: Skipped 1 previous similar message [ 6477.115612] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6477.149098] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6478.789459] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6480.063136] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6480.134734] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6481.453483] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6481.528271] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6482.400376] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6482.401084] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6482.403037] LustreError: Skipped 1 previous similar message [ 6482.403575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6482.406548] Lustre: Skipped 1 previous similar message [ 6482.410728] Lustre: Skipped 1 previous similar message [ 6487.521742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6487.523384] Lustre: Skipped 1 previous similar message [ 6488.237487] LustreError: 310965:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6488.240417] LustreError: 310965:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6488.308262] Lustre: server umount lustre-OST0000 complete [ 6491.615831] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6491.617582] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6491.621365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6495.638688] Lustre: server umount lustre-MDT0000 complete [ 6496.871055] LustreError: 308051:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753748965 with bad export cookie 651605352202673533 [ 6496.873695] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6496.874094] LustreError: 308051:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6496.935711] LustreError: 311369:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6496.938549] LustreError: 311369:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 6497.027888] Lustre: server umount lustre-MDT0001 complete [ 6499.928057] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 20:29:27 (1753748967) [ 6500.421162] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 6500.998452] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 20:29:28 (1753748968) [ 6501.987615] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 6504.630248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6505.923764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6507.229730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6508.582618] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6508.981867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6511.458577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6511.551196] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6511.559711] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6511.620798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6513.027758] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6516.012040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6516.042735] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6516.052564] Lustre: 313485:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6517.472268] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6519.278141] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6520.479490] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6522.568467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6522.587938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6522.643927] Lustre: Denying initial registration attempt from nid 192.168.202.105@tcp, specified as failover [ 6522.647598] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 6522.649938] LustreError: 315088:0:(tgt_mount.c:2272:server_fill_super()) Unable to start targets: -99 [ 6522.651962] LustreError: 315088:0:(tgt_mount.c:1766:server_put_super()) no obd lustre-OST0000 [ 6522.653471] LustreError: 315088:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 6522.704827] Lustre: server umount lustre-OST0000 complete [ 6522.706384] LustreError: 315088:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 6526.431863] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6526.434143] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6526.437419] Lustre: Skipped 1 previous similar message [ 6526.439695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6526.440992] Lustre: Skipped 2 previous similar messages [ 6538.207837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6538.209904] Lustre: Skipped 5 previous similar messages [ 6539.231056] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6539.260084] LustreError: 315457:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 6539.262575] LustreError: 315457:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6539.286719] Lustre: server umount lustre-MDT0000 complete [ 6540.460037] LustreError: 313476:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753749008 with bad export cookie 651605352202674415 [ 6540.463323] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6540.463907] LustreError: 313476:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6540.576408] Lustre: server umount lustre-MDT0001 complete [ 6545.139726] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 6546.167623] Key type lgssc unregistered [ 6546.304359] LNet: 316185:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6547.365276] LNet: Removed LNI 192.168.202.105@tcp [ 6547.627777] Key type .llcrypt unregistered [ 6547.628702] Key type ._llcrypt unregistered [ 6551.177127] Key type ._llcrypt registered [ 6551.178668] Key type .llcrypt registered [ 6551.217593] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 6551.520086] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6551.525907] alg: No test for adler32 (adler32-zlib) [ 6552.364748] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 6552.439197] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 6552.440801] LNet: Accept secure, port 988 [ 6554.015153] Key type lgssc registered [ 6554.347190] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6556.660141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6557.931624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6559.173546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6560.392603] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6560.727259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6567.077622] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 6570.016658] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6570.024392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6571.109974] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6571.116688] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6571.170082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6572.442696] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6575.263391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6575.296564] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6575.306540] Lustre: 318894:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6575.396578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6576.742235] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6578.605284] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6579.887618] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6582.047104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6582.069435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6582.127296] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6582.183365] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6583.994508] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6585.755443] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6586.438955] LustreError: 321149:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6586.493674] Lustre: server umount lustre-OST0000 complete [ 6589.935928] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6589.937197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6589.939962] Lustre: Skipped 1 previous similar message [ 6593.789128] LustreError: 321350:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 6593.792144] LustreError: 321350:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6593.816208] Lustre: server umount lustre-MDT0000 complete [ 6595.097684] LustreError: 319616:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753749063 with bad export cookie 8706381925161692770 [ 6595.101581] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6595.103646] LustreError: 319616:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6595.161854] LustreError: 321556:0:(obd_class.h:479:obd_check_dev()) Device 7 not setup [ 6595.163732] LustreError: 321556:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6595.258137] Lustre: server umount lustre-MDT0001 complete [ 6599.354724] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 6600.533551] Key type lgssc unregistered [ 6600.680375] LNet: 322082:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6601.701472] LNet: Removed LNI 192.168.202.105@tcp [ 6602.036844] Key type .llcrypt unregistered [ 6602.038193] Key type ._llcrypt unregistered [ 6611.400168] Key type ._llcrypt registered [ 6611.401589] Key type .llcrypt registered [ 6611.439481] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 6611.838658] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6611.848461] alg: No test for adler32 (adler32-zlib) [ 6612.703675] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 6612.789774] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 6612.791388] LNet: Accept secure, port 988 [ 6614.375169] Key type lgssc registered [ 6614.722202] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6617.286808] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6617.294055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6618.441085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6619.636366] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6622.483706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6622.603891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6623.919170] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6625.703250] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6626.849329] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6629.131739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6629.249566] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6631.174115] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6633.059121] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6648.287800] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6648.291406] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6648.297374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6648.801316] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6648.807506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6648.809691] Lustre: Skipped 1 previous similar message [ 6652.728515] LustreError: 325697:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6652.757030] Lustre: server umount lustre-MDT0000 complete [ 6653.915987] LustreError: 325699:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753749122 with bad export cookie 4675902694563329694 [ 6653.919094] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6653.919809] LustreError: 325699:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6653.923904] LustreError: 323442:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6653.924068] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6653.924073] Lustre: Skipped 1 previous similar message [ 6653.924655] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6653.924658] Lustre: Skipped 1 previous similar message [ 6653.937811] LustreError: 323442:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 6654.763398] LustreError: 323442:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6659.039813] LustreError: 323442:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6659.040059] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6659.048363] Lustre: Skipped 1 previous similar message [ 6664.159960] LustreError: 325586:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6664.160279] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6664.165165] LustreError: 325586:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 6664.166667] Lustre: Skipped 1 previous similar message [ 6668.255129] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6668.289036] LustreError: 325900:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6668.291649] LustreError: 325900:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6668.372544] Lustre: server umount lustre-MDT0001 complete [ 6670.442487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6673.195487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6679.519805] LustreError: 322760:0:(client.c:1385:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000e9c32700 x1838939196383104/t0(0) o250->MGC192.168.202.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6679.625320] LustreError: 325042:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6679.631370] LustreError: 325042:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 6679.651312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6679.669258] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 6681.020761] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6684.106753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6684.229293] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6685.548162] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6687.361773] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6694.374025] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 6697.095924] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 9 sec [ 6698.316106] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6698.882226] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6704.609227] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6704.609767] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6704.614021] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6704.619556] Lustre: Skipped 2 previous similar messages [ 6706.221092] LustreError: 328192:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6706.223912] LustreError: 328192:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6706.292743] Lustre: server umount lustre-OST0000 complete [ 6709.728621] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6709.729115] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6709.737061] Lustre: Skipped 2 previous similar messages [ 6713.595240] LustreError: 328393:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6713.597206] LustreError: 328393:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6713.624242] Lustre: server umount lustre-MDT0000 complete [ 6714.848131] LustreError: 326713:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6714.856036] LustreError: 326713:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 6714.883208] LustreError: 324158:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753749183 with bad export cookie 4675902694563336813 [ 6714.886350] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6714.888574] LustreError: 324158:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6715.034853] Lustre: server umount lustre-MDT0001 complete [ 6719.875681] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 6720.973613] Key type lgssc unregistered [ 6721.107410] LNet: 329122:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6722.149373] LNet: Removed LNI 192.168.202.105@tcp [ 6722.468319] Key type .llcrypt unregistered [ 6722.470055] Key type ._llcrypt unregistered [ 6731.653593] Key type ._llcrypt registered [ 6731.655031] Key type .llcrypt registered [ 6731.701231] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 6732.028934] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6732.057544] alg: No test for adler32 (adler32-zlib) [ 6732.927940] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 6733.009764] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 6733.011601] LNet: Accept secure, port 988 [ 6734.591185] Key type lgssc registered [ 6734.932048] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6737.624713] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6737.630870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6738.789200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6740.144288] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6743.045550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6743.178791] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6744.495174] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6746.244463] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6747.444469] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6748.442813] LustreError: 330516:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6748.445856] LustreError: 330516:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6748.448180] LustreError: 330516:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 6748.640433] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6748.640783] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6748.641211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6748.645674] Lustre: Skipped 1 previous similar message [ 6752.736257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6752.738051] Lustre: Skipped 2 previous similar messages [ 6754.555075] LustreError: 331864:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6754.580281] Lustre: server umount lustre-MDT0000 complete [ 6755.840182] LustreError: 330467:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753749224 with bad export cookie 7094931094149077159 [ 6755.842378] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6755.844118] LustreError: 330467:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6755.906207] LustreError: 332069:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6755.908519] LustreError: 332069:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6755.992091] Lustre: server umount lustre-MDT0001 complete [ 6760.778857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6760.872400] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6760.884255] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6760.955479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6762.220452] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6765.006031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6765.032866] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6765.042362] Lustre: 332839:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6765.129409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6766.392461] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6768.139659] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6769.339648] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6771.950978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6772.016155] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 6772.019756] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 6772.022451] LustreError: 334483:0:(tgt_mount.c:2272:server_fill_super()) Unable to start targets: -2 [ 6772.024513] LustreError: 334483:0:(tgt_mount.c:1766:server_put_super()) no obd lustre-OST0000 [ 6772.026465] LustreError: 334483:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 6772.073228] Lustre: server umount lustre-OST0000 complete [ 6772.074541] LustreError: 334483:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 6774.764412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6774.837628] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6774.911705] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6776.805484] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6778.635429] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6781.418298] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6781.447485] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6781.475062] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6781.502232] Lustre: lustre-OST0001: new disk, initializing [ 6781.503957] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6782.476909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6782.483086] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6782.508169] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 6782.537714] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6783.473374] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6785.274125] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6787.555713] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6787.557102] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6787.560133] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6787.562460] LustreError: Skipped 1 previous similar message [ 6788.332077] LustreError: 336583:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 6788.334629] LustreError: 336583:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6788.353898] Lustre: server umount lustre-OST0001 complete [ 6792.674159] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6792.674957] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6792.675867] LustreError: 336063:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6792.675879] LustreError: 336063:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 6792.676745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6792.676752] Lustre: Skipped 1 previous similar message [ 6792.694448] Lustre: Skipped 2 previous similar messages [ 6795.691062] LustreError: 336784:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6795.693017] LustreError: 336784:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6795.756814] Lustre: server umount lustre-OST0000 complete [ 6797.792207] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6797.792972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6797.796296] Lustre: Skipped 1 previous similar message [ 6797.798869] Lustre: Skipped 1 previous similar message [ 6803.199594] LustreError: 336985:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 6803.202062] LustreError: 336985:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6803.232662] Lustre: server umount lustre-MDT0000 complete [ 6804.541252] LustreError: 332831:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753749272 with bad export cookie 7094931094149077691 [ 6804.545304] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6804.545510] LustreError: 332831:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6804.685481] Lustre: server umount lustre-MDT0001 complete [ 6808.297867] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 6809.474549] Key type lgssc unregistered [ 6809.613619] LNet: 337712:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6810.661408] LNet: Removed LNI 192.168.202.105@tcp [ 6810.995820] Key type .llcrypt unregistered [ 6810.997422] Key type ._llcrypt unregistered [ 6822.444558] Key type ._llcrypt registered [ 6822.446247] Key type .llcrypt registered [ 6822.483233] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [ 6828.584014] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 6829.061119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6829.076804] alg: No test for adler32 (adler32-zlib) [ 6829.937481] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 6830.032064] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 6830.033906] LNet: Accept secure, port 988 [ 6831.615146] Key type lgssc registered [ 6831.982972] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6834.995657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6837.213861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6839.289750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6841.400499] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6843.692153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6843.707860] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6843.712658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6844.793477] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6844.804611] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6844.839411] Lustre: lustre-MDT0000: new disk, initializing [ 6844.864590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6844.872249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6846.207291] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6850.312506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6850.335292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6850.371860] Lustre: 341614:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6850.398260] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6850.400668] Lustre: Skipped 1 previous similar message [ 6850.438832] Lustre: lustre-MDT0001: new disk, initializing [ 6850.474504] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6850.489365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6850.493182] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6851.806166] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6854.830102] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6855.995055] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6858.307349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6858.333864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6858.434987] Lustre: lustre-OST0000: new disk, initializing [ 6858.436892] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6858.465876] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6860.390817] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6862.135641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6862.139255] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6862.167345] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6863.404963] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6864.753376] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6864.835824] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6866.114285] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6866.191575] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6867.427561] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6867.427983] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6867.432588] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6867.434131] LustreError: Skipped 1 previous similar message [ 6868.910094] LustreError: 344521:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6868.979943] Lustre: server umount lustre-OST0000 complete [ 6871.007899] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6871.010411] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6871.014178] Lustre: Skipped 1 previous similar message [ 6871.015799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6871.017738] Lustre: Skipped 1 previous similar message [ 6872.544310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6872.544937] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6876.349092] LustreError: 344722:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6876.351727] LustreError: 344722:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6876.383234] Lustre: server umount lustre-MDT0000 complete [ 6877.678613] LustreError: 341606:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753749346 with bad export cookie 10594312235274970148 [ 6877.682584] LustreError: 341606:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6877.683658] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6877.744665] LustreError: 344923:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6877.747377] LustreError: 344923:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6877.840514] Lustre: server umount lustre-MDT0001 complete [ 6880.632295] Lustre: DEBUG MARKER: == conf-sanity test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 20:35:48 (1753749348) [ 6882.677798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6884.797512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6892.524466] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [ 6895.630579] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 6899.320463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6901.736743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6903.949484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6906.222517] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6908.762521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6908.790963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6908.879260] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6908.894465] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6908.935308] Lustre: lustre-MDT0000: new disk, initializing [ 6908.969104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6908.975762] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6910.437525] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6914.792647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6914.819121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6914.852353] Lustre: 348675:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6914.876584] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6914.879045] Lustre: Skipped 1 previous similar message [ 6914.917936] Lustre: lustre-MDT0001: new disk, initializing [ 6914.951698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6914.967606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6914.972312] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6916.390576] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6919.665277] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6920.935880] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6923.340654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6923.371086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6923.467826] Lustre: lustre-OST0000: new disk, initializing [ 6923.470467] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6923.507907] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6923.700839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6923.704502] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6923.741820] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6925.458049] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6928.568253] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6929.893517] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6929.967097] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6931.205699] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6931.272786] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6933.986156] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6933.989043] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6933.989725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6933.992279] Lustre: Skipped 1 previous similar message [ 6933.993791] Lustre: Skipped 1 previous similar message [ 6939.105634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6939.108180] Lustre: Skipped 2 previous similar messages [ 6946.271116] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6946.291096] LustreError: 351580:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6946.294159] LustreError: 351580:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6946.367482] Lustre: server umount lustre-OST0000 complete [ 6949.344565] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6949.345362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6949.349759] Lustre: Skipped 1 previous similar message [ 6949.352230] Lustre: Skipped 2 previous similar messages [ 6953.722039] LustreError: 351783:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6953.723900] LustreError: 351783:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6953.752176] Lustre: server umount lustre-MDT0000 complete [ 6954.463841] LustreError: 348686:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6954.468549] LustreError: 348686:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 6954.958977] LustreError: 348667:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753749423 with bad export cookie 10594312235274971016 [ 6954.961791] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6954.962367] LustreError: 348667:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6955.088026] Lustre: server umount lustre-MDT0001 complete [ 6957.814633] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 20:37:05 (1753749425) [ 6960.179690] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 20:37:07 (1753749427) [ 6962.946609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6963.095565] LustreError: 352997:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6963.118248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6964.383483] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6967.145429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6968.608855] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6970.366098] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6971.564434] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6973.770869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6975.832639] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6977.561049] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6996.556094] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 6997.073853] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 6997.630687] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 6998.178285] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 6998.799716] Lustre: Failing over lustre-MDT0000 [ 6998.979332] LustreError: 355439:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6998.982057] LustreError: 355439:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6999.012476] Lustre: server umount lustre-MDT0000 complete [ 7003.104294] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7003.107182] LustreError: Skipped 1 previous similar message [ 7003.109029] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7003.114539] LustreError: 352997:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7008.555936] LustreError: 353701:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7008.562821] LustreError: 353701:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 7012.046035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7012.088297] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7012.176828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7012.180108] Lustre: Skipped 2 previous similar messages [ 7012.205683] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7013.507645] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7013.675532] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7017.444316] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 7017.451189] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 7017.467879] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7018.723208] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7019.280140] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7021.178785] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 7021.808843] Lustre: Failing over lustre-MDT0000 [ 7021.938462] LustreError: 356510:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7021.939986] LustreError: 356510:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7021.962882] Lustre: server umount lustre-MDT0000 complete [ 7022.561627] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7022.562392] LustreError: 352997:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7022.565619] Lustre: Skipped 4 previous similar messages [ 7022.572094] LustreError: 352997:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 7022.576943] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7032.800508] LustreError: 352997:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7032.806762] LustreError: 352997:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 8 previous similar messages [ 7034.916545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7034.960067] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7035.085780] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7036.504623] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7039.276120] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7040.485778] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 7040.487941] Lustre: Skipped 2 previous similar messages [ 7040.494231] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 7040.510962] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 7041.655423] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7042.211169] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7045.603331] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7045.603425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7045.608390] Lustre: Skipped 2 previous similar messages [ 7050.712629] Lustre: server umount lustre-MDT0000 complete [ 7050.722212] LustreError: 353015:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7050.729062] LustreError: 353015:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 7052.036473] LustreError: 352977:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753749520 with bad export cookie 10594312235274973683 [ 7052.040229] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7052.040287] LustreError: 352977:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7052.203101] Lustre: server umount lustre-MDT0001 complete [ 7062.768880] LustreError: 357937:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7062.771751] LustreError: 357937:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 7062.843249] Lustre: server umount lustre-OST0000 complete [ 7068.097073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7069.596162] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7072.539042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7073.968216] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7075.901543] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7077.261584] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7079.726988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7079.842936] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7079.845344] Lustre: Skipped 3 previous similar messages [ 7081.771987] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7083.624103] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7093.162702] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 7098.337121] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7098.338380] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7098.346187] Lustre: Skipped 3 previous similar messages [ 7098.349066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7098.352101] Lustre: Skipped 1 previous similar message [ 7101.369893] Lustre: server umount lustre-OST0000 complete [ 7108.816632] Lustre: server umount lustre-MDT0000 complete [ 7110.049054] LustreError: 358745:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753749578 with bad export cookie 10594312235274974390 [ 7110.051506] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7110.053841] LustreError: 358745:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7110.197123] Lustre: server umount lustre-MDT0001 complete [ 7115.173303] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 7116.367227] Key type lgssc unregistered [ 7116.513547] LNet: 361938:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7117.541427] LNet: Removed LNI 192.168.202.105@tcp [ 7117.848494] Key type .llcrypt unregistered [ 7117.849806] Key type ._llcrypt unregistered [ 7127.916217] Key type ._llcrypt registered [ 7127.917152] Key type .llcrypt registered [ 7127.961404] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 7128.423573] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7128.434443] alg: No test for adler32 (adler32-zlib) [ 7129.309130] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 7129.412944] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 7129.415876] LNet: Accept secure, port 988 [ 7131.015225] Key type lgssc registered [ 7131.439242] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7134.229990] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7134.236820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7135.401906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7136.727836] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7139.775044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7139.908028] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7141.251593] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7143.184865] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7144.509055] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7146.967037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7147.090658] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7149.198873] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7151.070586] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7161.451326] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 7166.944741] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7166.947027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7166.948673] Lustre: Skipped 2 previous similar messages [ 7166.950403] Lustre: Skipped 1 previous similar message [ 7171.708458] LustreError: 365658:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7171.738395] Lustre: server umount lustre-MDT0000 complete [ 7172.064282] LustreError: 363409:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7172.070437] LustreError: 363409:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 7173.031318] LustreError: 365559:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753749641 with bad export cookie 1549390326400497234 [ 7173.036056] LustreError: 365559:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7173.036698] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7173.106265] LustreError: 365860:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7173.108857] LustreError: 365860:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7173.200512] Lustre: server umount lustre-MDT0001 complete [ 7183.728955] LustreError: 366060:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7183.730765] LustreError: 366060:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7183.800096] Lustre: server umount lustre-OST0000 complete [ 7188.479520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7188.656808] LustreError: 366790:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7188.682997] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7190.081807] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7193.082021] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7193.197328] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7194.546826] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7196.445600] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7197.759898] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7200.160960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7202.144445] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7204.023236] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7215.656767] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 7228.896312] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7228.899888] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7228.904483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7228.906778] Lustre: Skipped 1 previous similar message [ 7230.945933] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7230.946898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7230.950052] Lustre: Skipped 1 previous similar message [ 7230.952621] Lustre: Skipped 1 previous similar message [ 7232.513189] LustreError: 369140:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7232.514940] LustreError: 369140:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7232.538187] Lustre: server umount lustre-MDT0000 complete [ 7233.748083] LustreError: 366771:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753749702 with bad export cookie 1549390326400498333 [ 7233.750264] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7233.753507] LustreError: 366771:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7233.893056] Lustre: server umount lustre-MDT0001 complete [ 7245.425187] LustreError: 369543:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7245.426966] LustreError: 369543:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 7245.494651] Lustre: server umount lustre-OST0000 complete [ 7250.878639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7251.058624] LustreError: 370370:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7251.082751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7251.085377] Lustre: Skipped 1 previous similar message [ 7252.464095] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7255.242520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7256.739650] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7258.628101] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7259.849140] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7262.035646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7263.831306] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7265.568445] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7277.478360] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 7282.657567] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7282.658312] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7282.658620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7282.658625] Lustre: Skipped 1 previous similar message [ 7282.662582] Lustre: Skipped 1 previous similar message [ 7287.777174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7287.779455] Lustre: Skipped 2 previous similar messages [ 7288.618122] LustreError: 372619:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7288.621065] LustreError: 372619:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7288.685585] Lustre: server umount lustre-OST0000 complete [ 7291.359868] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7291.363328] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7296.209313] Lustre: server umount lustre-MDT0000 complete [ 7297.388240] LustreError: 370350:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753749765 with bad export cookie 1549390326400499341 [ 7297.391311] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7297.391847] LustreError: 370350:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7297.533105] Lustre: server umount lustre-MDT0001 complete [ 7302.097638] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 7303.133345] Key type lgssc unregistered [ 7303.260416] LNet: 373549:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7304.293271] LNet: Removed LNI 192.168.202.105@tcp [ 7304.567887] Key type .llcrypt unregistered [ 7304.568915] Key type ._llcrypt unregistered [ 7313.886972] Key type ._llcrypt registered [ 7313.888159] Key type .llcrypt registered [ 7313.926222] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 7314.274669] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7314.325066] alg: No test for adler32 (adler32-zlib) [ 7315.205429] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 7315.303546] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 7315.305669] LNet: Accept secure, port 988 [ 7316.895165] Key type lgssc registered [ 7317.294830] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7320.051436] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7320.056710] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7320.059459] LustreError: 374948:0:(osd_handler.c:8372:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 7320.066925] LustreError: 374948:0:(obd_config.c:779:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 7320.069047] LustreError: 374948:0:(obd_mount.c:193:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 7320.071180] LustreError: 374948:0:(tgt_mount.c:2240:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 7320.073626] LustreError: 374948:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7322.588269] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7322.592126] LustreError: 375346:0:(osd_handler.c:8372:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 7322.601229] LustreError: 375346:0:(obd_config.c:779:class_setup()) setup lustre-OST0000-osd failed (-22) [ 7322.604150] LustreError: 375346:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 7322.606165] LustreError: 375346:0:(tgt_mount.c:2240:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 7322.609423] LustreError: 375346:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7329.091938] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 7330.252350] Key type lgssc unregistered [ 7330.399469] LNet: 376325:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7330.405476] LNet: Removed LNI 192.168.202.105@tcp [ 7330.742560] Key type .llcrypt unregistered [ 7330.743499] Key type ._llcrypt unregistered [ 7336.890890] Key type ._llcrypt registered [ 7336.892748] Key type .llcrypt registered [ 7336.946763] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [ 7343.200344] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 7343.606163] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7343.632422] alg: No test for adler32 (adler32-zlib) [ 7344.481129] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 7344.564930] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 7344.566362] LNet: Accept secure, port 988 [ 7346.143209] Key type lgssc registered [ 7346.486633] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7349.523029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7351.783890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7354.075928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7356.299986] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7358.610712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7358.633709] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7358.640418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7359.725893] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7359.737871] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7359.777640] Lustre: lustre-MDT0000: new disk, initializing [ 7359.805678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7359.812745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7361.094442] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7365.217115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7365.239290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7365.267994] Lustre: 379414:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7365.289037] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7365.290734] Lustre: Skipped 1 previous similar message [ 7365.326313] Lustre: lustre-MDT0001: new disk, initializing [ 7365.357859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7365.368623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7365.371701] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7366.632485] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7369.827372] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7371.061294] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7373.309694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7373.336147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7373.432657] Lustre: lustre-OST0000: new disk, initializing [ 7373.434793] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7373.465633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7375.432978] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7375.808725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7375.813572] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7375.846893] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7378.611439] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7379.982539] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7380.054371] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7381.410265] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7381.494234] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7384.173104] LustreError: 382319:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7384.239806] Lustre: server umount lustre-OST0000 complete [ 7386.083233] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7386.083311] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7386.084493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7386.086579] Lustre: Skipped 1 previous similar message [ 7391.201485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7391.203471] Lustre: Skipped 2 previous similar messages [ 7395.298447] LustreError: 382522:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7395.300179] LustreError: 382522:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7395.325994] Lustre: server umount lustre-MDT0000 complete [ 7396.528116] LustreError: 381145:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753749864 with bad export cookie 14421344793925834866 [ 7396.530129] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7396.530867] LustreError: 381145:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7406.051388] LustreError: 382724:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7406.054262] LustreError: 382724:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7406.139609] Lustre: server umount lustre-MDT0001 complete [ 7409.002485] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 7409.602490] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 20:44:37 (1753749877) [ 7411.914531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7412.096722] LustreError: 383444:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7412.105680] LustreError: 383444:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 7412.125754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7413.597797] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7416.583303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7416.698938] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7417.957777] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7419.789144] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7420.996453] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7423.300729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7425.218556] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7427.024747] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7429.433428] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7429.462075] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7429.527174] Lustre: lustre-OST0001: new disk, initializing [ 7429.529751] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7429.564937] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7429.568648] Lustre: Skipped 1 previous similar message [ 7431.594111] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7431.946043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7431.955455] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7432.002916] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7434.711080] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7437.284884] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7437.284966] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7437.285737] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7437.286693] LustreError: Skipped 1 previous similar message [ 7437.289608] Lustre: Skipped 1 previous similar message [ 7441.202721] Lustre: lustre-OST0001: Not available for connect from 192.168.202.5@tcp (stopping) [ 7441.206551] Lustre: Skipped 3 previous similar messages [ 7442.668103] LustreError: 386609:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 7442.670371] LustreError: 386609:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7442.688430] Lustre: server umount lustre-OST0001 complete [ 7443.428176] LustreError: 385026:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7443.435701] LustreError: 385026:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 7448.553810] LustreError: 385026:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7448.558102] LustreError: 385026:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 7453.669454] LustreError: 385026:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7453.673285] LustreError: 385026:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 7458.271831] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7458.274109] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7458.280113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7458.786435] LustreError: 385026:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7458.791699] LustreError: 385026:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 7460.781541] LustreError: 386834:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7460.783120] LustreError: 386834:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7460.841940] Lustre: server umount lustre-OST0000 complete [ 7462.879870] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7462.882395] LustreError: Skipped 1 previous similar message [ 7462.883653] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7462.887196] Lustre: Skipped 1 previous similar message [ 7468.309874] Lustre: server umount lustre-MDT0000 complete [ 7469.024295] LustreError: 386613:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7469.029059] LustreError: 386613:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 7469.528111] LustreError: 383425:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753749937 with bad export cookie 14421344793925835734 [ 7469.531909] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7469.533621] LustreError: 383425:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7475.643831] LustreError: 387238:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 7475.645803] LustreError: 387238:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 7475.737055] Lustre: server umount lustre-MDT0001 complete [ 7480.509111] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 7481.694376] Key type lgssc unregistered [ 7481.845397] LNet: 387764:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7482.917375] LNet: Removed LNI 192.168.202.105@tcp [ 7483.242605] Key type .llcrypt unregistered [ 7483.243486] Key type ._llcrypt unregistered [ 7494.490674] Key type ._llcrypt registered [ 7494.491869] Key type .llcrypt registered [ 7494.529759] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [ 7500.463916] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 7500.857506] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7500.871734] alg: No test for adler32 (adler32-zlib) [ 7501.737425] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 7501.826916] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 7501.829458] LNet: Accept secure, port 988 [ 7503.407153] Key type lgssc registered [ 7503.789743] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7506.811891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7509.045802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7511.007937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7513.104662] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7515.408323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7515.430456] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7515.436735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7516.526705] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7516.538309] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7516.580711] Lustre: lustre-MDT0000: new disk, initializing [ 7516.604650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7516.610561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7517.931885] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7522.097820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7522.119974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7522.151928] Lustre: 391670:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7522.178969] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7522.180701] Lustre: Skipped 1 previous similar message [ 7522.219551] Lustre: lustre-MDT0001: new disk, initializing [ 7522.256677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7522.271442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7522.275345] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7523.550962] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7526.641711] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7527.899274] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7530.114185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7530.138503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7530.224095] Lustre: lustre-OST0000: new disk, initializing [ 7530.226442] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7530.254613] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7532.157379] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7533.688102] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7533.690900] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7533.723852] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7535.333170] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7536.714324] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7536.810714] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7538.143037] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7538.213786] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7539.169200] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7539.170517] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7539.171614] LustreError: Skipped 1 previous similar message [ 7539.178871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7539.181049] Lustre: Skipped 1 previous similar message [ 7540.844043] LustreError: 394578:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7540.914366] Lustre: server umount lustre-OST0000 complete [ 7542.752074] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7542.755460] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7542.760940] Lustre: Skipped 1 previous similar message [ 7542.763224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7544.289071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7544.289715] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7548.283821] LustreError: 394780:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7548.286427] LustreError: 394780:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7548.312259] Lustre: server umount lustre-MDT0000 complete [ 7549.551110] LustreError: 391661:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753750017 with bad export cookie 6617001804620693858 [ 7549.554267] LustreError: 391661:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7549.555136] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7549.623564] LustreError: 394983:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7549.625645] LustreError: 394983:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7549.713565] Lustre: server umount lustre-MDT0001 complete [ 7552.536391] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 20:47:00 (1753750020) [ 7555.630051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7558.519155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7558.669095] Lustre: lustre-MDT0000: new disk, initializing [ 7558.680662] LustreError: 396247:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7558.688972] LustreError: 396247:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 7558.703665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7560.030307] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7562.976117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7563.096984] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7564.372661] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7566.221926] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7567.451210] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7568.118166] LustreError: 396275:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7568.121310] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7568.121816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7568.121822] Lustre: Skipped 1 previous similar message [ 7568.122873] LustreError: 396275:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7568.128387] Lustre: Skipped 1 previous similar message [ 7568.135772] LustreError: 396275:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 7573.280382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7573.282306] Lustre: Skipped 2 previous similar messages [ 7574.266818] LustreError: 397577:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7574.269609] LustreError: 397577:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7574.296697] Lustre: server umount lustre-MDT0000 complete [ 7575.484147] LustreError: 396227:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753750043 with bad export cookie 6617001804620694733 [ 7575.487579] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7575.488110] LustreError: 396227:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7575.627049] Lustre: server umount lustre-MDT0001 complete [ 7578.875939] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 20:47:26 (1753750046) [ 7581.124432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7581.274566] LustreError: 398498:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7581.298321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7582.727523] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7585.774333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7587.295992] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7589.135399] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7590.389186] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7592.699412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7594.702210] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7596.537650] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7609.857480] Lustre: Permanently deactivating lustre-OST0000 [ 7609.865336] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7617.007926] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7618.349725] LustreError: 401237:0:(mgs_llog.c:1937:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 7618.354112] LustreError: 401237:0:(mgs_llog.c:1991:mgs_replace_nids()) Only MGS is allowed to be started [ 7618.356673] LustreError: 401237:0:(mgs_handler.c:1032:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7618.998898] LustreError: 401285:0:(mgs_llog.c:1991:mgs_replace_nids()) Only MGS is allowed to be started [ 7619.002316] LustreError: 401285:0:(mgs_handler.c:1032:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7625.710045] LustreError: 401383:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7625.712330] LustreError: 401383:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 7625.780820] Lustre: server umount lustre-OST0000 complete [ 7626.705845] LustreError: 401534:0:(mgs_llog.c:1991:mgs_replace_nids()) Only MGS is allowed to be started [ 7626.707753] LustreError: 401534:0:(mgs_handler.c:1032:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7631.840688] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7631.840776] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7631.841614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7631.844343] Lustre: Skipped 1 previous similar message [ 7636.963564] LustreError: 401632:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7636.965170] LustreError: 401632:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7636.988354] Lustre: server umount lustre-MDT0000 complete [ 7638.285588] LustreError: 401634:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753750106 with bad export cookie 6617001804620695265 [ 7638.288354] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7638.291035] LustreError: 401634:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7647.791092] Lustre: server umount lustre-MDT0001 complete [ 7650.765893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7652.085579] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7652.993703] LustreError: 402671:0:(mgs_handler.c:1032:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 7655.085187] Lustre: 403010:0:(mgs_llog.c:1698:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.105@tcp [ 7655.766625] LustreError: 403108:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7655.768543] LustreError: 403108:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 7655.793234] Lustre: server umount MGS complete [ 7659.608747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7659.751913] LustreError: 403735:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7659.758232] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7659.762126] Lustre: Skipped 1 previous similar message [ 7659.783091] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7659.786438] Lustre: Skipped 2 previous similar messages [ 7661.091455] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7663.947351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7664.050077] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7665.383213] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7667.212017] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7668.339396] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7668.956284] Lustre: Permanently reactivating lustre-OST0000 [ 7668.962068] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7668.964021] Lustre: Skipped 2 previous similar messages [ 7694.303168] LustreError: 405276:0:(osp_dev.c:694:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 7697.226347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7697.339988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7697.343448] Lustre: Skipped 1 previous similar message [ 7699.245323] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7701.129756] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7713.761493] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7713.761711] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7713.763955] LustreError: Skipped 1 previous similar message [ 7713.767783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7713.770799] Lustre: Skipped 4 previous similar messages [ 7714.989106] LustreError: 407099:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7714.992151] LustreError: 407099:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 7715.049367] Lustre: server umount lustre-OST0000 complete [ 7718.419348] Lustre: server umount lustre-MDT0000 complete [ 7718.881561] LustreError: 403734:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7718.890323] LustreError: 403734:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 7719.594165] LustreError: 405069:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753750187 with bad export cookie 6617001804620696210 [ 7719.595301] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7719.596851] LustreError: 405069:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7719.602527] Lustre: 403726:0:(mgc_request.c:1876:mgc_process_log()) MGC192.168.202.105@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 7719.743139] Lustre: server umount lustre-MDT0001 complete [ 7724.456872] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 7725.566581] Key type lgssc unregistered [ 7725.710401] LNet: 408025:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7726.757367] LNet: Removed LNI 192.168.202.105@tcp [ 7727.103850] Key type .llcrypt unregistered [ 7727.106087] Key type ._llcrypt unregistered [ 7733.353408] Key type ._llcrypt registered [ 7733.354364] Key type .llcrypt registered [ 7733.395694] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [ 7739.915734] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 7740.248994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7740.260820] alg: No test for adler32 (adler32-zlib) [ 7741.111602] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 7741.195381] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 7741.197686] LNet: Accept secure, port 988 [ 7742.775119] Key type lgssc registered [ 7743.123730] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7746.054380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7748.238536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7750.184801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7752.342471] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7754.582078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7754.604121] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7754.609530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7755.693470] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7755.704823] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7755.739432] Lustre: lustre-MDT0000: new disk, initializing [ 7755.763427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7755.772134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7757.076528] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7761.220718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7761.248930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7761.289223] Lustre: 411111:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7761.316830] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7761.320394] Lustre: Skipped 1 previous similar message [ 7761.366359] Lustre: lustre-MDT0001: new disk, initializing [ 7761.408893] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7761.421244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7761.424228] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7762.738741] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7765.879824] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7767.135750] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7769.342054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7769.367772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7769.481122] Lustre: lustre-OST0000: new disk, initializing [ 7769.483194] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7769.516073] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7771.311292] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7773.180931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7773.183761] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7773.219351] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7774.295778] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7775.574321] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7775.658717] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7776.966201] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7777.047237] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7778.276953] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7778.277201] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7778.281264] LustreError: Skipped 1 previous similar message [ 7778.285748] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7778.289730] Lustre: Skipped 1 previous similar message [ 7779.757064] LustreError: 414020:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7779.832631] Lustre: server umount lustre-OST0000 complete [ 7781.856169] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7781.858277] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7781.861211] Lustre: Skipped 1 previous similar message [ 7781.862813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7783.393606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7783.394278] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7787.198092] LustreError: 414220:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7787.199974] LustreError: 414220:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7787.240603] Lustre: server umount lustre-MDT0000 complete [ 7788.417227] LustreError: 411103:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753750256 with bad export cookie 12349499761858185530 [ 7788.420783] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7788.488852] LustreError: 414421:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7788.490585] LustreError: 414421:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7788.576959] Lustre: server umount lustre-MDT0001 complete [ 7791.237474] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 7791.740983] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 20:50:59 (1753750259) [ 7793.798275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7793.961902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7795.104194] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7797.651042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7798.894424] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7800.579751] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7801.783013] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7804.015407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7804.132863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7804.134644] Lustre: Skipped 1 previous similar message [ 7805.978616] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7807.734603] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7808.394973] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 7824.243581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 7828.960047] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7828.963704] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7828.970118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7828.972919] Lustre: Skipped 1 previous similar message [ 7831.723117] LustreError: 417581:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7831.724798] LustreError: 417581:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7831.791439] Lustre: server umount lustre-OST0000 complete [ 7833.567895] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7833.569835] LustreError: Skipped 1 previous similar message [ 7833.570969] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7833.573705] Lustre: Skipped 1 previous similar message [ 7833.575312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7833.576769] Lustre: Skipped 2 previous similar messages [ 7845.344129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7845.345857] Lustre: Skipped 5 previous similar messages [ 7847.391105] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7847.430145] LustreError: 417782:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7847.432572] LustreError: 417782:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7847.459659] Lustre: server umount lustre-MDT0000 complete [ 7848.655752] LustreError: 415121:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753750317 with bad export cookie 12349499761858186398 [ 7848.659111] LustreError: 415121:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 7848.659735] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7848.804961] Lustre: server umount lustre-MDT0001 complete [ 7853.537554] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 7854.619757] Key type lgssc unregistered [ 7854.751598] LNet: 418513:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7855.781513] LNet: Removed LNI 192.168.202.105@tcp [ 7856.094455] Key type .llcrypt unregistered [ 7856.095736] Key type ._llcrypt unregistered [ 7863.856784] Key type ._llcrypt registered [ 7863.857784] Key type .llcrypt registered [ 7863.898983] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 7863.998291] Key type .llcrypt unregistered [ 7863.999399] Key type ._llcrypt unregistered [ 7870.862156] Key type ._llcrypt registered [ 7870.863099] Key type .llcrypt registered [ 7870.892480] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 7871.188265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7871.233065] alg: No test for adler32 (adler32-zlib) [ 7872.075533] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 7872.150740] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 7872.152209] LNet: Accept secure, port 988 [ 7873.727101] Key type lgssc registered [ 7874.031070] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7876.492411] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7876.498787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7877.658709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7878.893275] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7881.533899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7881.652207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7883.406057] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7885.124403] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7886.817403] LustreError: 420645:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7886.821683] LustreError: 420645:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 7887.350567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7887.468367] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7888.487446] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7888.711819] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7892.448150] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7892.450141] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7892.453428] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7893.985727] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7893.986265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7893.989824] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7897.515111] LustreError: 422679:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 7897.580129] Lustre: server umount lustre-OST0000 complete [ 7899.104753] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7899.105512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7899.108023] Lustre: Skipped 1 previous similar message [ 7899.109469] Lustre: Skipped 1 previous similar message [ 7904.224068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7904.225644] Lustre: Skipped 2 previous similar messages [ 7904.826807] LustreError: 422881:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7904.828524] LustreError: 422881:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7904.853716] Lustre: server umount lustre-MDT0000 complete [ 7906.026199] LustreError: 420625:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753750374 with bad export cookie 17801358428962762843 [ 7906.028497] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7906.029159] LustreError: 420625:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7906.090300] LustreError: 423083:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7906.091948] LustreError: 423083:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7906.162995] Lustre: server umount lustre-MDT0001 complete [ 7910.351527] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 7911.331165] Key type lgssc unregistered [ 7911.458336] LNet: 423609:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7912.485470] LNet: Removed LNI 192.168.202.105@tcp [ 7912.796229] Key type .llcrypt unregistered [ 7912.798088] Key type ._llcrypt unregistered [ 7921.484348] Key type ._llcrypt registered [ 7921.485540] Key type .llcrypt registered [ 7921.527143] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 7921.907154] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7921.926174] alg: No test for adler32 (adler32-zlib) [ 7922.795524] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 7922.884326] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 7922.886065] LNet: Accept secure, port 988 [ 7924.463123] Key type lgssc registered [ 7924.780134] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7927.292314] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7927.297789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7944.799304] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7950.943217] LustreError: 424944:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.105@tcp: failed processing log, type 4: rc = -110 [ 7980.639130] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7982.411022] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7984.108177] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7986.277390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7986.462484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7986.483365] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 7987.749394] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7990.521545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7990.639972] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7991.833862] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7993.590560] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7994.757893] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7998.947256] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7998.951863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7998.954713] Lustre: Skipped 1 previous similar message [ 8003.053103] LustreError: 427211:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 8003.130250] Lustre: server umount lustre-OST0000 complete [ 8006.111967] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8006.114034] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8006.117210] Lustre: Skipped 1 previous similar message [ 8006.118824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8009.184848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8009.185256] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8010.493870] LustreError: 427413:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 8010.495674] LustreError: 427413:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8010.525177] Lustre: server umount lustre-MDT0000 complete [ 8011.684207] LustreError: 426568:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753750480 with bad export cookie 14064566188384251084 [ 8011.687139] LustreError: 426568:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8011.687527] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8011.754062] LustreError: 427616:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8011.755728] LustreError: 427616:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8011.837830] Lustre: server umount lustre-MDT0001 complete [ 8016.199665] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 8017.314090] Key type lgssc unregistered [ 8017.445315] LNet: 428142:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8018.469400] LNet: Removed LNI 192.168.202.105@tcp [ 8018.755200] Key type .llcrypt unregistered [ 8018.756741] Key type ._llcrypt unregistered [ 8027.625086] Key type ._llcrypt registered [ 8027.626388] Key type .llcrypt registered [ 8027.660905] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 8027.955053] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8027.974022] alg: No test for adler32 (adler32-zlib) [ 8028.818863] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 8028.894735] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 8028.896250] LNet: Accept secure, port 988 [ 8030.471120] Key type lgssc registered [ 8030.800502] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8033.371397] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8033.378477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8034.531871] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8035.863534] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 8039.209175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8039.322948] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8040.519548] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 8042.279455] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8043.483600] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8045.665064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8045.775677] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8047.543289] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 8049.161561] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8059.115927] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 8064.484401] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8064.487265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8064.493235] Lustre: Skipped 1 previous similar message [ 8064.499523] Lustre: Skipped 2 previous similar messages [ 8066.700220] LustreError: 431751:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8066.736086] Lustre: server umount lustre-MDT0000 complete [ 8067.825472] LustreError: 429484:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.5@tcp arrived at 1753750536 with bad export cookie 3815084524718934606 [ 8067.831949] LustreError: 429484:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8068.699099] LustreError: 429484:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753750537 with bad export cookie 3815084524718933955 [ 8068.703083] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8068.704810] LustreError: 429484:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8069.600148] LustreError: 429503:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8069.600648] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8069.601134] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8069.601146] Lustre: Skipped 1 previous similar message [ 8069.602117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8069.602129] Lustre: Skipped 1 previous similar message [ 8069.603882] LustreError: 429503:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 8073.631909] LustreError: 429503:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8073.632304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8073.635904] LustreError: 429503:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 8074.795965] LustreError: 431954:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8074.798463] LustreError: 431954:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8074.857787] Lustre: server umount lustre-OST0000 complete [ 8076.912902] LustreError: 432307:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 8076.916114] LustreError: 432307:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8076.997488] Lustre: server umount lustre-MDT0001 complete [ 8081.651830] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 8082.784430] Key type lgssc unregistered [ 8082.909634] LNet: 432834:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8083.941290] LNet: Removed LNI 192.168.202.105@tcp [ 8084.248722] Key type .llcrypt unregistered [ 8084.251121] Key type ._llcrypt unregistered [ 8093.172243] Key type ._llcrypt registered [ 8093.173117] Key type .llcrypt registered [ 8093.206399] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 8093.538992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8093.602393] alg: No test for adler32 (adler32-zlib) [ 8094.443369] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 8094.520738] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 8094.522156] LNet: Accept secure, port 988 [ 8096.095136] Key type lgssc registered [ 8096.390929] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8098.864043] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8098.869926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8100.021221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8101.351000] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 8104.394262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8104.523652] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8105.834131] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 8107.683781] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8108.977632] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8111.290442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8111.432054] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8113.341318] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 8115.133264] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8124.779919] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 8130.020648] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8130.021050] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8130.023272] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8130.026206] Lustre: Skipped 1 previous similar message [ 8130.867517] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.5@tcp (stopping) [ 8130.869188] Lustre: Skipped 1 previous similar message [ 8132.736482] LustreError: 436447:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 8132.756436] Lustre: server umount lustre-MDT0001 complete [ 8135.137768] LustreError: 434199:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8135.138666] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8135.139071] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8135.140146] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8135.141686] LustreError: 434199:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 8140.256132] LustreError: 434198:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8140.256623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8140.397276] LustreError: 436650:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8140.399099] LustreError: 436650:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8140.458158] Lustre: server umount lustre-OST0000 complete [ 8141.696024] LustreError: 436851:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8141.698082] LustreError: 436851:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8141.782392] Lustre: server umount lustre-MDT0000 complete [ 8147.317386] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 8148.322824] Key type lgssc unregistered [ 8148.452385] LNet: 437530:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8149.477353] LNet: Removed LNI 192.168.202.105@tcp [ 8149.750629] Key type .llcrypt unregistered [ 8149.751557] Key type ._llcrypt unregistered [ 8158.069359] Key type ._llcrypt registered [ 8158.070367] Key type .llcrypt registered [ 8158.102950] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [ 8163.758127] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 8164.069143] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8164.108290] alg: No test for adler32 (adler32-zlib) [ 8164.950775] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 8165.024967] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 8165.026327] LNet: Accept secure, port 988 [ 8166.599100] Key type lgssc registered [ 8166.894815] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8169.584760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8171.662766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8173.583768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8175.542405] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8178.332229] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 8181.353194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8183.404551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8183.422936] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8183.427487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8184.511669] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8184.522704] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8184.563637] Lustre: lustre-MDT0000: new disk, initializing [ 8184.588155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8184.595560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8185.774926] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 8189.091479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8191.113218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8191.133279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8191.223139] Lustre: lustre-OST0000: new disk, initializing [ 8191.224866] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8191.227033] Lustre: Skipped 1 previous similar message [ 8191.246932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8192.970208] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 8195.852767] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8196.603643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 8196.606157] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 8196.612889] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 8203.954140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8205.980571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8206.002974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8206.032152] Lustre: 441448:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8206.055779] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8206.090410] Lustre: lustre-MDT0001: new disk, initializing [ 8206.129683] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8206.143335] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 8206.146476] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 8207.306087] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 8210.666526] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 8210.728519] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 8211.888688] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 8211.957685] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 8216.545839] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8216.546108] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8216.549482] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8216.553709] Lustre: Skipped 1 previous similar message [ 8216.556646] Lustre: Skipped 1 previous similar message [ 8217.394173] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.5@tcp (stopping) [ 8219.579675] LustreError: 444835:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8219.605383] Lustre: server umount lustre-MDT0001 complete [ 8226.785224] LustreError: 441454:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8226.789052] LustreError: 441454:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 8231.904912] LustreError: 441460:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8231.905408] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8231.915601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8237.024418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8237.025107] LustreError: 441460:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8242.144441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8242.145202] LustreError: 441460:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8246.239117] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8246.253783] LustreError: 445108:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 8246.255746] LustreError: 445108:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8246.315178] Lustre: server umount lustre-OST0000 complete [ 8247.264873] LustreError: 441460:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8253.693169] LustreError: 445310:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 8253.694762] LustreError: 445310:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8253.776029] Lustre: server umount lustre-MDT0000 complete [ 8259.030808] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 8260.027922] Key type lgssc unregistered [ 8260.152375] LNet: 445989:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8261.221320] LNet: Removed LNI 192.168.202.105@tcp [ 8261.521098] Key type .llcrypt unregistered [ 8261.522172] Key type ._llcrypt unregistered [ 8274.250724] Key type ._llcrypt registered [ 8274.252562] Key type .llcrypt registered [ 8274.289589] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 8274.389303] Key type .llcrypt unregistered [ 8274.390254] Key type ._llcrypt unregistered [ 8280.803443] Key type ._llcrypt registered [ 8280.804392] Key type .llcrypt registered [ 8280.836581] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 8281.228794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8281.234695] alg: No test for adler32 (adler32-zlib) [ 8282.071745] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 8282.149973] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 8282.151410] LNet: Accept secure, port 988 [ 8283.727118] Key type lgssc registered [ 8284.054316] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8286.661456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8288.837073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8291.028904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8293.106380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8293.122649] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8293.127958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8294.210829] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8294.220931] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8294.255541] Lustre: lustre-MDT0000: new disk, initializing [ 8294.279161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8294.285055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8295.487084] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 8299.397799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8299.425829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8299.458625] Lustre: 449291:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8299.490884] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8299.492262] Lustre: Skipped 1 previous similar message [ 8299.532985] Lustre: lustre-MDT0001: new disk, initializing [ 8299.571667] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8299.584613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8299.588973] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8300.783721] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 8303.705563] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8304.858465] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8307.059880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8307.084307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8307.174422] Lustre: lustre-OST0000: new disk, initializing [ 8307.176304] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8307.203823] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8308.969758] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 8309.370516] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8309.373431] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8309.406069] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8311.915901] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8314.853321] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8314.854917] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8314.855945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8314.861224] Lustre: Skipped 2 previous similar messages [ 8319.969356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8319.975876] Lustre: Skipped 4 previous similar messages [ 8320.263480] LustreError: 451708:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8320.301725] Lustre: server umount lustre-MDT0000 complete [ 8321.730226] LustreError: 449281:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753750790 with bad export cookie 593829784604995798 [ 8321.739160] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8321.740730] LustreError: 449281:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8321.826439] LustreError: 451912:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8321.830422] LustreError: 451912:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8321.925882] Lustre: server umount lustre-MDT0001 complete [ 8333.616437] LustreError: 452114:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 8333.618378] LustreError: 452114:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8333.688471] Lustre: server umount lustre-OST0000 complete [ 8337.279100] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 21:00:04 (1753750804) [ 8337.608727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8339.941905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8340.109135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8341.404148] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 8344.650459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8344.790681] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8346.081603] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 8348.343544] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8349.825673] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8352.902878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8352.938183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8353.025588] Lustre: Found index 0 for lustre-OST0000, updating log [ 8353.091554] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8355.401397] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 8357.658544] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8365.420802] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 8370.658280] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8370.659826] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8370.663988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8370.665628] LustreError: Skipped 1 previous similar message [ 8370.667489] Lustre: Skipped 1 previous similar message [ 8372.017292] LustreError: 455201:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8372.022094] LustreError: 455201:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8372.093474] Lustre: server umount lustre-OST0000 complete [ 8373.157443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8375.776383] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8375.781095] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8375.788546] Lustre: Skipped 1 previous similar message [ 8375.792559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8375.796507] Lustre: Skipped 2 previous similar messages [ 8376.027149] LustreError: 455456:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 8376.030117] LustreError: 455456:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8376.047510] Lustre: server umount lustre-MDT0000 complete [ 8376.801597] LustreError: 452941:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8376.810177] LustreError: 452941:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 8377.825115] LustreError: 452941:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8377.833131] LustreError: 452941:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 8377.849432] LustreError: 452920:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753750846 with bad export cookie 593829784604997177 [ 8377.853222] LustreError: 452920:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8377.853605] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8378.004770] Lustre: server umount lustre-MDT0001 complete [ 8381.250480] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 21:00:48 (1753750848) [ 8381.590764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8381.960659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8382.376255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8382.773444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8383.168671] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8383.564220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8384.014701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8384.394246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8384.787428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8385.132347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8385.505276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8385.890287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8386.231066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8386.607723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8386.993547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8387.510941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8387.895921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8388.389072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8388.902863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8389.240241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8389.563607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8390.010376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8390.425531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8390.910958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8391.297137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8391.643171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8392.023553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8392.425711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8392.842251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8393.333705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8393.803653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8394.298435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8394.680771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8395.011433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8395.347060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8395.680671] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8396.046703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8396.409557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8396.745045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8397.074347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8397.423085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8397.850224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8398.222136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8398.568177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8398.890133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8399.204783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8399.522159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8399.839567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8400.173387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8400.509064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8400.846950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8404.183342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8404.354429] LustreError: 459049:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8404.381791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8406.068808] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 8409.290783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8410.735946] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 8412.614495] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8413.952995] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8416.458826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8416.495325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8416.590168] Lustre: Found index 0 for lustre-OST0000, updating log [ 8418.891960] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 8420.730520] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8424.931258] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8424.935282] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8424.946135] Lustre: Skipped 1 previous similar message [ 8424.950268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8427.627137] LustreError: 461321:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8427.628928] LustreError: 461321:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 8427.685488] Lustre: server umount lustre-OST0000 complete [ 8429.029703] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8429.034769] Lustre: Skipped 1 previous similar message [ 8433.120290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8433.122056] Lustre: Skipped 9 previous similar messages [ 8443.359139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8443.431705] Lustre: server umount lustre-MDT0000 complete [ 8444.384919] LustreError: 459049:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8444.394357] LustreError: 459049:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 8445.239979] LustreError: 459074:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753750913 with bad export cookie 593829784604998248 [ 8445.246219] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8445.247819] LustreError: 459074:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8445.342050] LustreError: 461729:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 8445.345241] LustreError: 461729:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 8445.432376] Lustre: server umount lustre-MDT0001 complete [ 8446.765121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8453.951463] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [ 8458.109606] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 8461.637230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8463.965380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8466.474546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8468.855272] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8471.296067] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 21:02:18 (1753750938) [ 8490.079428] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 8491.066182] Key type lgssc unregistered [ 8491.195409] LNet: 465288:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8615.143185] LNet: Removed LNI 192.168.202.105@tcp [ 8615.530028] Key type .llcrypt unregistered [ 8615.531374] Key type ._llcrypt unregistered [ 8621.890134] Key type ._llcrypt registered [ 8621.891047] Key type .llcrypt registered [ 8621.925647] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 8622.320972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8622.382353] alg: No test for adler32 (adler32-zlib) [ 8623.240182] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 8623.324719] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 8623.326205] LNet: Accept secure, port 988 [ 8624.903142] Key type lgssc registered [ 8625.239295] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8626.776670] LNet: Added LNI 192.168.202.105@tcp100 [8/256/0/180] [ 8627.280672] Lustre: DEBUG MARKER: SKIP: conf-sanity test_73c unable to configure net [ 8634.435415] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 8635.779023] Key type lgssc unregistered [ 8635.950256] LNet: 467139:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8635.952325] LNet: Removed LNI 192.168.202.105@tcp [ 8635.958538] LNet: Removed LNI 192.168.202.105@tcp100 [ 8636.233471] Key type .llcrypt unregistered [ 8636.234425] Key type ._llcrypt unregistered [ 8644.381798] Key type ._llcrypt registered [ 8644.382791] Key type .llcrypt registered [ 8644.416318] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [ 8650.999212] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 8651.260194] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8651.266772] alg: No test for adler32 (adler32-zlib) [ 8652.100269] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 8652.171833] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 8652.173209] LNet: Accept secure, port 988 [ 8653.743191] Key type lgssc registered [ 8654.223226] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8658.034808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8660.436825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8662.990803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8665.839290] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8668.743311] Lustre: DEBUG MARKER: == conf-sanity test 74: Test per-device adaptive timeout parameters ========================================================== 21:05:36 (1753751136) [ 8671.987205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8672.008938] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8672.013320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8673.105882] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8673.125199] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8673.176372] Lustre: lustre-MDT0000: new disk, initializing [ 8673.200741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8673.207449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8674.784778] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 8679.476736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8679.511529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8679.561117] Lustre: 470612:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8679.591391] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8679.594158] Lustre: Skipped 1 previous similar message [ 8679.632437] Lustre: lustre-MDT0001: new disk, initializing [ 8679.671525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8679.688030] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8679.691582] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8681.220484] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 8684.980350] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8686.745705] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8689.954352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8689.981971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8690.093289] Lustre: lustre-OST0000: new disk, initializing [ 8690.095156] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8690.124915] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8691.401465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8691.410653] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8691.463530] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8692.422299] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 8696.297220] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8701.580670] loop0: detected capacity change from 0 to 204800000 [ 8701.660120] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8701.662866] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8701.665866] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8701.669333] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8701.692208] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8701.720587] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8701.739508] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8701.772719] blk_update_request: operation not supported error, dev loop0, sector 176912 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8701.806468] blk_update_request: operation not supported error, dev loop0, sector 185112 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8701.844654] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8704.354862] loop0: detected capacity change from 0 to 204800000 [ 8704.497028] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8704.577832] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8706.743684] loop0: detected capacity change from 0 to 204800000 [ 8708.647529] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8708.682684] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8708.749462] Lustre: fs15246-OST0000: new disk, initializing [ 8708.752828] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 8708.783589] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8710.981951] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 8715.392071] loop1: detected capacity change from 0 to 204800000 [ 8717.667535] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8717.702632] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8717.742861] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 8717.773995] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 8717.821063] Lustre: fs15246-MDT0000: new disk, initializing [ 8717.885273] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8717.899276] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8717.959567] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 8719.008798] Lustre: 475159:0:(client.c:1611:after_reply()) @@@ resending request on EINPROGRESS req@00000000a2af8a55 x1838941334443904/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1753751203 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 8719.404840] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 8728.097480] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 8728.101440] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 8728.109243] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 8728.591519] LustreError: 476274:0:(obd_class.h:479:obd_check_dev()) Device 30 not setup [ 8728.615446] Lustre: server umount fs15246-MDT0000 complete [ 8730.086785] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8738.034152] LustreError: 476775:0:(obd_class.h:479:obd_check_dev()) Device 32 not setup [ 8738.038875] LustreError: 476775:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8738.065185] Lustre: server umount fs15246-OST0000 complete [ 8742.881591] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8742.881620] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8742.885884] LustreError: Skipped 1 previous similar message [ 8742.893248] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8742.897395] Lustre: Skipped 1 previous similar message [ 8745.443819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8745.446100] Lustre: Skipped 1 previous similar message [ 8748.715076] LustreError: 477277:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8748.716665] LustreError: 477277:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 8748.780488] Lustre: server umount lustre-OST0000 complete [ 8750.561423] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8750.562344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8750.573543] Lustre: Skipped 2 previous similar messages [ 8755.680978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8755.684541] Lustre: Skipped 2 previous similar messages [ 8756.362990] LustreError: 477480:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8756.365524] LustreError: 477480:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8756.398849] Lustre: server umount lustre-MDT0000 complete [ 8757.760682] LustreError: 470602:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753751226 with bad export cookie 4803464781389268244 [ 8757.764309] LustreError: 470602:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8757.765531] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8757.920129] Lustre: server umount lustre-MDT0001 complete [ 8763.183714] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 8764.581518] Key type lgssc unregistered [ 8764.716381] LNet: 478207:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8765.733741] LNet: Removed LNI 192.168.202.105@tcp [ 8766.224765] Key type .llcrypt unregistered [ 8766.227116] Key type ._llcrypt unregistered [ 8775.095255] Key type ._llcrypt registered [ 8775.096574] Key type .llcrypt registered [ 8775.131659] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 8775.609628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8775.616186] alg: No test for adler32 (adler32-zlib) [ 8776.491535] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 8776.587482] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 8776.589041] LNet: Accept secure, port 988 [ 8778.175174] Key type lgssc registered [ 8778.766881] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8782.385306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8784.555965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8786.969645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8789.162269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8796.220362] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [ 8800.321714] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 8804.337090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8807.359685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8810.071893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8812.769891] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8815.822144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8815.846841] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8815.852833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8816.949687] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8816.966684] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8817.006179] Lustre: lustre-MDT0000: new disk, initializing [ 8817.033284] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8817.040715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8818.463910] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 8823.294486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8823.325626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8823.372880] Lustre: 482583:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8823.395219] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8823.397709] Lustre: Skipped 1 previous similar message [ 8823.447191] Lustre: lustre-MDT0001: new disk, initializing [ 8823.492456] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8823.510402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8823.515275] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8825.078766] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 8828.407609] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8829.763569] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8832.526091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8832.560140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8832.704808] Lustre: lustre-OST0000: new disk, initializing [ 8832.707959] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8832.755344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8834.879606] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 8838.101986] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8838.136334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8838.141577] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8838.174505] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8839.494810] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8839.560656] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8840.984705] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8841.100961] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8843.234469] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8843.235459] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8843.238395] LustreError: Skipped 1 previous similar message [ 8843.239379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8843.248416] Lustre: Skipped 1 previous similar message [ 8843.947827] LustreError: 485488:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8844.002665] Lustre: server umount lustre-OST0000 complete [ 8848.354063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8848.355523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8848.360577] Lustre: Skipped 1 previous similar message [ 8848.365451] Lustre: Skipped 2 previous similar messages [ 8851.577686] LustreError: 485690:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8851.580283] LustreError: 485690:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8851.611455] Lustre: server umount lustre-MDT0000 complete [ 8852.963256] LustreError: 482573:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753751321 with bad export cookie 329721825246368426 [ 8852.967835] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8852.969831] LustreError: 482573:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8853.048717] LustreError: 485892:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8853.052849] LustreError: 485892:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8853.134659] Lustre: server umount lustre-MDT0001 complete [ 8862.095941] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [ 8865.499967] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 8869.310495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8871.898322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8874.257032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8876.480183] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8877.039963] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8624 sec ========== 21:09:04 (1753751344) [ 8877.687546] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 21:09:05 (1753751345) === [ 8878.876252] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 21:09:06 (1753751346) === [ 8892.827175] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 8894.125237] Key type lgssc unregistered [ 8894.277726] LNet: 490719:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8895.333421] LNet: Removed LNI 192.168.202.105@tcp [ 8895.628172] Key type .llcrypt unregistered [ 8895.629136] Key type ._llcrypt unregistered