[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 494330606 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002340] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008012] pid_max: default: 32768 minimum: 301 [ 0.009130] LSM: Security Framework initializing [ 0.010055] Yama: becoming mindful. [ 0.011037] SELinux: Initializing. [ 0.012073] *** VALIDATE selinux *** [ 0.020687] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024703] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025156] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026105] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027115] *** VALIDATE tmpfs *** [ 0.028458] *** VALIDATE proc *** [ 0.030052] *** VALIDATE cgroup *** [ 0.031010] *** VALIDATE cgroup2 *** [ 0.033186] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034139] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036029] Spectre V2 : User space: Vulnerable [ 0.037007] Speculative Store Bypass: Vulnerable [ 0.040316] debug: unmapping init [mem 0xffffffff89459000-0xffffffff89460fff] [ 0.042496] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044314] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045019] ... version: 2 [ 0.046009] ... bit width: 48 [ 0.047010] ... generic registers: 4 [ 0.048014] ... value mask: 0000ffffffffffff [ 0.049016] ... max period: 00007fffffffffff [ 0.050016] ... fixed-purpose events: 3 [ 0.051014] ... event mask: 000000070000000f [ 0.052296] rcu: Hierarchical SRCU implementation. [ 0.054384] smp: Bringing up secondary CPUs ... [ 0.055580] x86: Booting SMP configuration: [ 0.056023] .... node #0, CPUs: #1 #2 #3 [ 0.059449] smp: Brought up 1 node, 4 CPUs [ 0.061014] smpboot: Max logical packages: 1 [ 0.062017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.135562] node 0 deferred pages initialised in 71ms [ 0.139007] devtmpfs: initialized [ 0.140252] x86/mm: Memory block size: 128MB [ 0.142801] gcov: version magic: 0x41383552 [ 0.146170] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.149116] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.151274] pinctrl core: initialized pinctrl subsystem [ 0.153160] [ 0.153752] ************************************************************* [ 0.157013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.159010] ** ** [ 0.161012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.164011] ** ** [ 0.166010] ** This means that this kernel is built to expose internal ** [ 0.168010] ** IOMMU data structures, which may compromise security on ** [ 0.170011] ** your system. ** [ 0.172011] ** ** [ 0.174013] ** If you see this message and you are not debugging the ** [ 0.177013] ** kernel, report this immediately to your vendor! ** [ 0.179009] ** ** [ 0.181011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.184010] ************************************************************* [ 0.186558] NET: Registered protocol family 16 [ 0.188380] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.190057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.193059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.196441] cpuidle: using governor menu [ 0.197619] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.200414] PCI: Using configuration type 1 for base access [ 0.202119] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.210072] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.212039] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.216126] cryptd: max_cpu_qlen set to 1000 [ 0.219280] ACPI: Added _OSI(Module Device) [ 0.221029] ACPI: Added _OSI(Processor Device) [ 0.222014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.224013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.227333] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.234165] ACPI: Interpreter enabled [ 0.235054] ACPI: PM: (supports S0 S3 S4 S5) [ 0.237014] ACPI: Using IOAPIC for interrupt routing [ 0.238182] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.242408] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.250924] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.253051] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.255018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.258073] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.263199] acpiphp: Slot [2] registered [ 0.264105] acpiphp: Slot [3] registered [ 0.265110] acpiphp: Slot [4] registered [ 0.267092] acpiphp: Slot [5] registered [ 0.268109] acpiphp: Slot [6] registered [ 0.269098] acpiphp: Slot [7] registered [ 0.271122] acpiphp: Slot [8] registered [ 0.272090] acpiphp: Slot [9] registered [ 0.273126] acpiphp: Slot [10] registered [ 0.275102] acpiphp: Slot [11] registered [ 0.276079] acpiphp: Slot [12] registered [ 0.277108] acpiphp: Slot [13] registered [ 0.279083] acpiphp: Slot [14] registered [ 0.280104] acpiphp: Slot [15] registered [ 0.282089] acpiphp: Slot [16] registered [ 0.283111] acpiphp: Slot [17] registered [ 0.285087] acpiphp: Slot [18] registered [ 0.286098] acpiphp: Slot [19] registered [ 0.287085] acpiphp: Slot [20] registered [ 0.289090] acpiphp: Slot [21] registered [ 0.290082] acpiphp: Slot [22] registered [ 0.291113] acpiphp: Slot [23] registered [ 0.293097] acpiphp: Slot [24] registered [ 0.294089] acpiphp: Slot [25] registered [ 0.296100] acpiphp: Slot [26] registered [ 0.297092] acpiphp: Slot [27] registered [ 0.298093] acpiphp: Slot [28] registered [ 0.300088] acpiphp: Slot [29] registered [ 0.301113] acpiphp: Slot [30] registered [ 0.303096] acpiphp: Slot [31] registered [ 0.304052] PCI host bridge to bus 0000:00 [ 0.306017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.308024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.310020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.312024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.315032] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.318030] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.320193] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.323178] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.328402] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.339016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.344055] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.346022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.348016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.354020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.356446] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.359744] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.362042] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.365488] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.370000] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.385818] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.391013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.398021] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.404019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.409016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.422023] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.430729] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.440016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.447016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.462015] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.472233] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.478015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.486014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.506025] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.521454] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.528017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.534016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.554015] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.563483] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.568014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.576019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.590021] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.599545] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.606015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.613017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.627028] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.637000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.641389] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.643388] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.646371] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.648226] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.653028] iommu: Default domain type: Passthrough [ 0.654402] SCSI subsystem initialized [ 0.656137] ACPI: bus type USB registered [ 0.657102] usbcore: registered new interface driver usbfs [ 0.659075] usbcore: registered new interface driver hub [ 0.660073] usbcore: registered new device driver usb [ 0.661144] pps_core: LinuxPPS API ver. 1 registered [ 0.662013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.665053] PTP clock support registered [ 0.667057] EDAC MC: Ver: 3.0.0 [ 0.668501] PCI: Using ACPI for IRQ routing [ 0.670847] NetLabel: Initializing [ 0.672011] NetLabel: domain hash size = 128 [ 0.674010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.676079] NetLabel: unlabeled traffic allowed by default [ 0.678123] vgaarb: loaded [ 0.679275] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.681015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.689000] clocksource: Switched to clocksource kvm-clock [ 0.796946] VFS: Disk quotas dquot_6.6.0 [ 0.798615] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.801186] *** VALIDATE ramfs *** [ 0.802213] *** VALIDATE hugetlbfs *** [ 0.803896] pnp: PnP ACPI init [ 0.806973] pnp: PnP ACPI: found 6 devices [ 0.826889] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.830469] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.832518] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.834471] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.836925] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.839409] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.842337] NET: Registered protocol family 2 [ 0.844702] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.848896] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.852204] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.856982] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.860348] TCP: Hash tables configured (established 65536 bind 65536) [ 0.862810] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.865428] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.868335] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.871411] NET: Registered protocol family 1 [ 0.874407] RPC: Registered named UNIX socket transport module. [ 0.876488] RPC: Registered udp transport module. [ 0.877961] RPC: Registered tcp transport module. [ 0.879769] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.882052] NET: Registered protocol family 44 [ 0.883574] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.885336] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.887377] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.889495] PCI: CLS 0 bytes, default 64 [ 0.891186] Unpacking initramfs... [ 2.290958] debug: unmapping init [mem 0xffff8f59fcc54000-0xffff8f59fffbffff] [ 2.295323] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.297474] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.300699] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.785366] Initialise system trusted keyrings [ 2.787860] Key type blacklist registered [ 2.790360] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.800221] zbud: loaded [ 2.803606] *** VALIDATE nfs *** [ 2.805250] *** VALIDATE nfs4 *** [ 2.806873] pstore: using deflate compression [ 2.809899] Platform Keyring initialized [ 2.919645] NET: Registered protocol family 38 [ 2.921363] Key type asymmetric registered [ 2.922780] Asymmetric key parser 'x509' registered [ 2.924711] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.927644] io scheduler mq-deadline registered [ 2.929175] io scheduler kyber registered [ 2.930724] io scheduler bfq registered [ 2.932793] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.935569] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.938523] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.941420] ACPI: Power Button [PWRF] [ 3.033046] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.128258] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.311674] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.401305] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.580954] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.614786] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.645798] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.651192] Non-volatile memory driver v1.3 [ 3.652970] Linux agpgart interface v0.103 [ 3.682559] virtio_blk virtio1: [vda] 133640 512-byte logical blocks (68.4 MB/65.3 MiB) [ 3.685461] vda: detected capacity change from 0 to 68423680 [ 3.703199] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.706399] vdb: detected capacity change from 0 to 1073741824 [ 3.718901] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.721500] vdc: detected capacity change from 0 to 2621440000 [ 3.734721] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.737762] vdd: detected capacity change from 0 to 2621440000 [ 3.752625] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.755644] vde: detected capacity change from 0 to 4294967296 [ 3.769951] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.772560] vdf: detected capacity change from 0 to 4294967296 [ 3.780682] libphy: Fixed MDIO Bus: probed [ 3.786226] usbcore: registered new interface driver usbserial_generic [ 3.788831] usbserial: USB Serial support registered for generic [ 3.791276] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.795749] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.797484] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.799914] mousedev: PS/2 mouse device common for all mice [ 3.803064] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.804403] rtc_cmos 00:05: RTC can wake from S4 [ 3.809545] rtc_cmos 00:05: registered as rtc0 [ 3.812327] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.812604] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.815893] intel_pstate: CPU model not supported [ 3.822371] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.822555] hid: raw HID events driver (C) Jiri Kosina [ 3.829423] usbcore: registered new interface driver usbhid [ 3.831575] usbhid: USB HID core driver [ 3.835532] drop_monitor: Initializing network drop monitor service [ 3.838292] Initializing XFRM netlink socket [ 3.840651] NET: Registered protocol family 10 [ 3.849644] Segment Routing with IPv6 [ 3.850685] NET: Registered protocol family 17 [ 3.853654] mpls_gso: MPLS GSO support [ 3.860760] RAS: Correctable Errors collector initialized. [ 3.863197] AVX version of gcm_enc/dec engaged. [ 3.864855] AES CTR mode by8 optimization enabled [ 3.946635] sched_clock: Marking stable (3946615212, 0)->(4856321397, -909706185) [ 3.949199] registered taskstats version 1 [ 3.950798] Loading compiled-in X.509 certificates [ 3.952296] zswap: loaded using pool lzo/zbud [ 3.975642] Key type big_key registered [ 3.987067] Key type encrypted registered [ 3.988599] ima: No TPM chip found, activating TPM-bypass! [ 3.990466] ima: Allocated hash algorithm: sha1 [ 3.991973] ima: No architecture policies found [ 3.993530] evm: Initialising EVM extended attributes: [ 3.995166] evm: security.selinux [ 3.996256] evm: security.ima [ 3.997223] evm: security.capability [ 3.998388] evm: HMAC attrs: 0x1 [ 4.001271] rtc_cmos 00:05: setting system clock to 2025-11-14 20:47:34 UTC (1763153254) [ 4.013591] debug: unmapping init [mem 0xffffffff8a403000-0xffffffff8a5fffff] [ 4.016690] debug: unmapping init [mem 0xffffffff89182000-0xffffffff89458fff] [ 4.025119] Write protecting the kernel read-only data: 28672k [ 4.028429] debug: unmapping init [mem 0xffffffff87803000-0xffffffff879fffff] [ 4.031203] debug: unmapping init [mem 0xffffffff88114000-0xffffffff881fffff] [ 4.064504] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.072913] systemd[1]: Detected virtualization kvm. [ 4.074680] systemd[1]: Detected architecture x86-64. [ 4.076648] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.106138] systemd[1]: No hostname configured. [ 4.107697] systemd[1]: Set hostname to . [ 4.109726] random: systemd: uninitialized urandom read (16 bytes read) [ 4.112142] systemd[1]: Initializing machine ID from random generator. [ 4.147461] random: ln: uninitialized urandom read (6 bytes read) [ 4.236592] random: systemd: uninitialized urandom read (16 bytes read) [ 4.239696] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.244535] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 4.252815] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Reached target Swap. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.832087] device-mapper: uevent: version 1.0.3 [ 4.834375] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.568216] random: fast init done [ 5.570095] virtio_net virtio0 ens2: renamed from eth0 [ 5.603776] scsi host0: ata_piix [ 5.646195] scsi host1: ata_piix [ 5.691282] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.694038] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.854898] dracut-initqueue[581]: RTNETLINK answers: File exists [ 10.515899] random: crng init done [ 10.517352] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.879655] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 14.197832] printk: systemd: 26 output lines suppressed due to ratelimiting [ 14.829947] SELinux: Disabled at runtime. [ 14.936867] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 14.943408] systemd[1]: Detected virtualization kvm. [ 14.945281] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 16.095770] systemd[1]: initrd-switch-root.service: Succeeded. [ 16.099397] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 16.109807] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 16.118421] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 16.130333] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 16.151763] systemd[1]: Starting Journal Service... Starting Journal Service... [ 16.165082] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ 16.421871] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Initrd File Systems. Starting Remount Root and Kernel File Systems... Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 17.264288] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 18.083679] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 18.107861] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 18.384382] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 18.459807] EDAC sbridge: Ver: 1.1.2 [ 21.620715] Key type dns_resolver registered [ 22.289601] NFS: Registering the id_resolver key type [ 22.304429] Key type id_resolver registered [ 22.309459] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (6s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg322-server login: [ 73.926655] libcfs: loading out-of-tree module taints kernel. [ 73.977995] Key type ._llcrypt registered [ 73.979564] Key type .llcrypt registered [ 74.084877] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 90.418567] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 92.197274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 92.211794] alg: No test for adler32 (adler32-zlib) [ 93.614929] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 94.687324] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 96.479157] Key type lgssc registered [ 98.354040] Lustre: Echo OBD driver; http://www.lustre.org/ [ 100.023895] hrtimer: interrupt took 7591491 ns [ 115.226607] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 115.981722] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 123.926323] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 131.640105] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 138.761985] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 154.791470] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 168.055600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 168.128183] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 168.162457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 169.461691] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 169.507320] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 169.642639] Lustre: lustre-MDT0000: new disk, initializing [ 169.737069] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 169.767767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 173.631349] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 186.571895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 186.632770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 186.679157] Lustre: 6512:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 186.695046] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 186.701106] Lustre: Skipped 1 previous similar message [ 186.785390] Lustre: lustre-MDT0001: new disk, initializing [ 186.802074] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 186.860950] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 186.914074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 186.932236] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 190.494475] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 194.765823] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 202.892349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 202.947360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 203.120619] Lustre: lustre-OST0000: new disk, initializing [ 203.126441] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 203.168071] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 205.797977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 205.808188] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 205.899833] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 208.384441] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 221.499533] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 221.588287] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 221.731987] Lustre: lustre-OST0001: new disk, initializing [ 221.735646] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 221.806341] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 226.577031] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 230.451256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 230.466387] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 230.529915] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 237.294542] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 243.763298] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 254.074291] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing check_logdir /tmp/testlogs/ [ 258.000769] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing yml_node [ 262.621515] Lustre: DEBUG MARKER: Client: 2.16.61.2 [ 265.321226] Lustre: DEBUG MARKER: MDS: 2.16.61.2 [ 267.996803] Lustre: DEBUG MARKER: OSS: 2.16.61.2 [ 269.921740] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Nov 14 15:51:58 EST 2025 [ 288.068694] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 289.937145] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 307.170033] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 307.174101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 307.196941] Lustre: Skipped 3 previous similar messages [ 307.202227] Lustre: Skipped 2 previous similar messages [ 311.963396] LustreError: 12220:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 312.075571] Lustre: server umount lustre-MDT0000 complete [ 315.372433] LustreError: 6505:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763153565 with bad export cookie 2742604868051339774 [ 315.375473] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 315.389014] LustreError: 6505:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 317.407911] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 317.411821] LustreError: 6520:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 317.412154] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 317.429214] Lustre: Skipped 1 previous similar message [ 317.485446] LustreError: 6520:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 320.419990] LustreError: 8016:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 320.424309] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 320.436303] LustreError: 8016:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 320.446280] Lustre: Skipped 2 previous similar messages [ 321.573382] LustreError: 12422:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 321.579409] LustreError: 12422:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 321.746852] Lustre: server umount lustre-MDT0001 complete [ 325.448623] LustreError: 12623:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 325.459013] LustreError: 12623:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 325.556195] Lustre: server umount lustre-OST0000 complete [ 329.014964] LustreError: 12824:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 329.017541] LustreError: 12824:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 329.121909] Lustre: server umount lustre-OST0001 complete [ 334.454697] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 342.818984] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 350.944446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 357.117493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 363.083661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 368.865655] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 375.175982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 375.232420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 375.386459] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 375.412961] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 375.459577] Lustre: lustre-MDT0000: new disk, initializing [ 375.528156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 375.546043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 379.093623] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 387.832967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 387.881122] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 387.939676] Lustre: 15012:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 387.991290] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 387.995233] Lustre: Skipped 1 previous similar message [ 388.049768] Lustre: lustre-MDT0001: new disk, initializing [ 388.132569] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 388.161864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 388.168627] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 391.189927] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 397.698320] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 400.942976] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 407.797175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 407.892402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 408.216886] Lustre: lustre-OST0000: new disk, initializing [ 408.225770] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 408.391950] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 409.485958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 409.506519] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 409.630132] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 413.641964] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 420.333974] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 424.073362] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 424.385556] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 427.942936] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 428.152874] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 430.053537] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 430.055418] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 430.056978] LustreError: Skipped 1 previous similar message [ 430.057738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 430.065108] Lustre: Skipped 1 previous similar message [ 435.769839] LustreError: 17950:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 435.780812] LustreError: 17950:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 435.881788] Lustre: server umount lustre-OST0000 complete [ 440.290343] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 440.313462] Lustre: Skipped 1 previous similar message [ 440.314495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 440.334892] Lustre: Skipped 4 previous similar messages [ 445.712714] LustreError: 18151:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 445.723247] LustreError: 18151:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 445.929995] Lustre: server umount lustre-MDT0000 complete [ 449.921712] LustreError: 15793:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763153700 with bad export cookie 2742604868051341307 [ 449.926308] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 449.947603] LustreError: 15793:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 450.555322] Lustre: server umount lustre-MDT0001 complete [ 455.967818] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 15:55:03 (1763153703) [ 458.886839] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 460.955820] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 15:55:09 (1763153709) [ 467.925674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 468.466631] LustreError: 19022:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 468.538546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 472.302700] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 473.571768] LustreError: 19022:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 477.670863] LustreError: 19022:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 480.493507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 484.470355] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 489.620678] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 492.502161] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 498.949663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 504.480716] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 509.624501] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 522.211265] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 522.215965] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 522.242295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 522.251463] Lustre: Skipped 2 previous similar messages [ 523.241736] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 527.615361] LustreError: 21281:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 527.618027] LustreError: 21281:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 527.730552] Lustre: server umount lustre-OST0000 complete [ 531.939216] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 531.942743] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 531.979933] Lustre: Skipped 1 previous similar message [ 537.448290] Lustre: server umount lustre-MDT0000 complete [ 538.593489] LustreError: 19022:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 538.604758] LustreError: 19022:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 540.294506] LustreError: 19003:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763153790 with bad export cookie 2742604868051342189 [ 540.296252] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 540.326992] LustreError: 19003:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 540.690923] Lustre: server umount lustre-MDT0001 complete [ 549.957504] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 552.523982] Key type lgssc unregistered [ 552.801195] LNet: 22217:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 552.809075] LNetError: 22217:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 552.826693] LNet: Removed LNI 192.168.203.122@tcp [ 553.560126] Key type .llcrypt unregistered [ 553.564486] Key type ._llcrypt unregistered [ 574.220583] Key type ._llcrypt registered [ 574.222253] Key type .llcrypt registered [ 574.292112] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 575.053627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 575.159505] alg: No test for adler32 (adler32-zlib) [ 576.155899] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 576.384364] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 578.031170] Key type lgssc registered [ 578.926988] Lustre: Echo OBD driver; http://www.lustre.org/ [ 585.733633] loop: module loaded [ 585.764110] loop0: detected capacity change from 0 to 204800000 [ 585.890265] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 585.895845] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 586.118473] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 586.170156] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 591.004379] loop0: detected capacity change from 0 to 204800000 [ 591.170499] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 591.185339] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 591.206384] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 591.330394] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 595.940528] loop0: detected capacity change from 0 to 204800000 [ 601.188283] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 601.239165] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 601.250679] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 602.456939] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 602.498915] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 602.546501] Lustre: lustre-MDT0000: new disk, initializing [ 602.636624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 602.647808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 606.230746] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 614.466268] loop1: detected capacity change from 0 to 204800000 [ 619.078455] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 619.134620] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 619.362453] Lustre: lustre-OST07c6: new disk, initializing [ 619.371437] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 619.374519] Lustre: Skipped 1 previous similar message [ 619.439582] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 623.845598] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 628.834775] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 628.844563] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 628.881561] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 633.279650] Lustre: 24377:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 633.289071] Lustre: 24377:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 633.295780] Lustre: 24377:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 633.304826] Lustre: 24377:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 633.316798] Lustre: 24377:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 633.334155] Lustre: 24377:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 645.805372] Lustre: 24376:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 645.811836] Lustre: 24376:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 645.815800] Lustre: 24376:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 645.825967] Lustre: 24376:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 645.833246] Lustre: 24376:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 645.837957] Lustre: 24376:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 650.499169] Lustre: 24376:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 650.506962] Lustre: 24376:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 1 previous similar message [ 650.518375] Lustre: 24376:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 650.527752] Lustre: 24376:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 1 previous similar message [ 650.535363] Lustre: 24376:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 650.545283] Lustre: 24376:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 1 previous similar message [ 650.552539] Lustre: 24376:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 650.565498] Lustre: 24376:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 1 previous similar message [ 650.569615] Lustre: 24376:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 650.572621] Lustre: 24376:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 1 previous similar message [ 650.575601] Lustre: 24376:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 650.578843] Lustre: 24376:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 653.736065] Lustre: 24376:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 653.749252] Lustre: 24376:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 1 previous similar message [ 653.763000] Lustre: 24376:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 653.769706] Lustre: 24376:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 1 previous similar message [ 653.779387] Lustre: 24376:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 653.793190] Lustre: 24376:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 1 previous similar message [ 653.800325] Lustre: 24376:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 653.805987] Lustre: 24376:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 1 previous similar message [ 653.812832] Lustre: 24376:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 653.819310] Lustre: 24376:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 1 previous similar message [ 653.826240] Lustre: 24376:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 653.830242] Lustre: 24376:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 658.278434] Lustre: 24375:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 658.285982] Lustre: 24375:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 1 previous similar message [ 658.290904] Lustre: 24375:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 658.295738] Lustre: 24375:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 1 previous similar message [ 658.299994] Lustre: 24375:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 658.308035] Lustre: 24375:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 1 previous similar message [ 658.314649] Lustre: 24375:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 658.320995] Lustre: 24375:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 1 previous similar message [ 658.327328] Lustre: 24375:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 658.332504] Lustre: 24375:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 1 previous similar message [ 658.341847] Lustre: 24375:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 658.346647] Lustre: 24375:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 668.889439] Lustre: 24376:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 668.901025] Lustre: 24376:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 5 previous similar messages [ 668.909052] Lustre: 24376:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 668.912761] Lustre: 24376:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 668.918025] Lustre: 24376:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 668.930146] Lustre: 24376:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 668.937928] Lustre: 24376:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 668.954116] Lustre: 24376:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 668.959037] Lustre: 24376:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 668.966819] Lustre: 24376:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 668.974419] Lustre: 24376:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 668.978154] Lustre: 24376:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 715.746320] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 715.759207] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 715.777313] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 716.667156] LustreError: 26562:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 716.752686] Lustre: server umount lustre-OST07c6 complete [ 724.237742] LustreError: 27063:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 724.241142] LustreError: 27063:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 724.398957] Lustre: server umount lustre-MDT0000 complete [ 739.042526] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 741.606967] Key type lgssc unregistered [ 741.843580] LNet: 27896:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 741.852451] LNetError: 27896:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 741.874383] LNet: Removed LNI 192.168.203.122@tcp [ 742.616950] Key type .llcrypt unregistered [ 742.619499] Key type ._llcrypt unregistered [ 757.262172] Key type ._llcrypt registered [ 757.264684] Key type .llcrypt registered [ 757.353201] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 758.087832] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 758.206499] alg: No test for adler32 (adler32-zlib) [ 759.197305] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 759.379926] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 761.031163] Key type lgssc registered [ 762.003741] Lustre: Echo OBD driver; http://www.lustre.org/ [ 768.883171] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 768.895638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 770.424510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 774.123172] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 779.756667] LustreError: 29125:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 779.762562] LustreError: 29125:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 781.920239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 782.205342] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 785.635117] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 790.315787] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 793.461952] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 799.522753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 799.976407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 805.498433] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 810.276715] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 815.340248] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 819.794858] LustreError: 31335:0:(qsd_reint.c:618:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:2 until pending updates are flushed. [ 830.947613] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 830.948065] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 830.975839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 830.983106] Lustre: Skipped 1 previous similar message [ 834.265130] LustreError: 31587:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 834.365237] Lustre: server umount lustre-OST0000 complete [ 838.626855] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 838.632074] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 838.647869] Lustre: Skipped 1 previous similar message [ 838.650433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 841.193108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 841.195748] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 843.445929] LustreError: 31789:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 843.457588] LustreError: 31789:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 843.587040] Lustre: server umount lustre-MDT0000 complete [ 846.305103] LustreError: 31500:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 846.319970] LustreError: 31500:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 846.576566] LustreError: 29105:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763154097 with bad export cookie 4621115653734759036 [ 846.592067] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 846.687041] LustreError: 31992:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 846.689882] LustreError: 31992:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 846.929149] Lustre: server umount lustre-MDT0001 complete [ 855.136331] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 857.117805] Key type lgssc unregistered [ 857.306070] LNet: 32522:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 857.311273] LNetError: 32522:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 857.327268] LNet: Removed LNI 192.168.203.122@tcp [ 857.958752] Key type .llcrypt unregistered [ 857.960142] Key type ._llcrypt unregistered [ 873.193534] Key type ._llcrypt registered [ 873.201651] Key type .llcrypt registered [ 873.294763] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 874.004734] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 874.046620] alg: No test for adler32 (adler32-zlib) [ 875.187290] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 875.452677] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 877.111988] Key type lgssc registered [ 878.238787] Lustre: Echo OBD driver; http://www.lustre.org/ [ 885.556626] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 885.572428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 886.906562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 889.992309] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 896.475966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 896.743635] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 900.076663] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 905.198764] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 908.372428] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 915.207728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 915.540549] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 920.798487] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 925.337954] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 932.986940] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 943.584214] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 943.592491] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 943.601090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 943.606873] Lustre: Skipped 1 previous similar message [ 948.710808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 948.716097] Lustre: Skipped 2 previous similar messages [ 949.190104] LustreError: 36022:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 949.282985] Lustre: server umount lustre-OST0000 complete [ 953.316201] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 953.319309] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 953.344138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 958.444239] LustreError: 36224:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 958.447755] LustreError: 36224:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 958.530232] Lustre: server umount lustre-MDT0000 complete [ 958.945105] LustreError: 34536:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 958.958169] LustreError: 34536:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 962.124222] LustreError: 33732:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763154212 with bad export cookie 2265891693106542642 [ 962.125428] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 962.154540] LustreError: 33732:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 962.272576] LustreError: 36425:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 962.284107] LustreError: 36425:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 962.430549] Lustre: server umount lustre-MDT0001 complete [ 972.566458] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 975.286965] Key type lgssc unregistered [ 975.585733] LNet: 36959:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 975.599155] LNetError: 36959:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 975.615524] LNet: Removed LNI 192.168.203.122@tcp [ 976.287028] Key type .llcrypt unregistered [ 976.289340] Key type ._llcrypt unregistered [ 995.395958] Key type ._llcrypt registered [ 995.401141] Key type .llcrypt registered [ 995.485868] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 996.025923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 996.039311] alg: No test for adler32 (adler32-zlib) [ 997.115502] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 997.331734] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 998.975179] Key type lgssc registered [ 1000.012207] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1006.790381] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1006.803963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1008.130914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1011.111342] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1018.339050] LustreError: 38333:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1018.356558] LustreError: 38333:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1019.467342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1019.793812] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1023.517853] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1028.533814] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1031.228273] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1037.079958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1037.333397] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1041.443974] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1045.286702] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1052.740570] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1060.836775] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1060.840710] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1060.863868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1062.501834] LustreError: 40603:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1062.617766] Lustre: server umount lustre-MDT0000 complete [ 1062.889626] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1062.891967] LustreError: 39367:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1062.956011] LustreError: 39367:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1066.194698] LustreError: 38329:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1066.573925] LustreError: 39954:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763154317 with bad export cookie 8899896054046079442 [ 1066.578092] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1066.603851] LustreError: 39954:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1068.006178] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1068.047303] Lustre: Skipped 1 previous similar message [ 1068.061515] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1068.064554] Lustre: Skipped 1 previous similar message [ 1071.010066] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1071.010068] LustreError: 38329:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1071.010073] LustreError: 38329:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1072.849605] LustreError: 40807:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1072.858094] LustreError: 40807:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1072.974236] Lustre: server umount lustre-MDT0001 complete [ 1092.480317] LustreError: 41028:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1092.482846] LustreError: 41028:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1092.560431] Lustre: server umount lustre-OST0000 complete [ 1104.663353] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 1106.955452] Key type lgssc unregistered [ 1107.219071] LNet: 41867:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1107.230786] LNetError: 41867:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1108.263804] LNet: Removed LNI 192.168.203.122@tcp [ 1108.857036] Key type .llcrypt unregistered [ 1108.858764] Key type ._llcrypt unregistered [ 1126.197184] Key type ._llcrypt registered [ 1126.200342] Key type .llcrypt registered [ 1126.286906] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 1126.815258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1126.856765] alg: No test for adler32 (adler32-zlib) [ 1127.840210] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 1127.989873] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 1129.607148] Key type lgssc registered [ 1130.431450] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1136.441211] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1136.477727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1137.834377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1140.565366] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1146.459503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1146.779886] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1149.711107] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1154.092580] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1157.890129] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1163.705963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1164.047412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1169.240494] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1175.094547] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1183.542192] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1192.181805] Lustre: lustre-OST0000: Not available for connect from 192.168.203.22@tcp (stopping) [ 1192.929160] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1192.939205] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1192.953770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1193.955684] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1193.962347] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1197.138565] LustreError: 45509:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1197.288329] Lustre: server umount lustre-OST0000 complete [ 1218.528588] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1218.544813] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1218.567271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1218.569790] Lustre: Skipped 1 previous similar message [ 1220.937586] LustreError: 45885:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1220.940242] LustreError: 45885:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1221.135100] Lustre: server umount lustre-MDT0000 complete [ 1224.783225] LustreError: 43223:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763154475 with bad export cookie 2232747903600662572 [ 1224.785367] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1224.799616] LustreError: 43223:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1230.963835] LustreError: 46087:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1230.988411] LustreError: 46087:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1231.214853] Lustre: server umount lustre-MDT0001 complete [ 1242.431710] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 1245.362730] Key type lgssc unregistered [ 1245.647645] LNet: 46620:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1245.670905] LNetError: 46620:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1245.696642] LNet: Removed LNI 192.168.203.122@tcp [ 1246.462830] Key type .llcrypt unregistered [ 1246.464305] Key type ._llcrypt unregistered [ 1268.928109] Key type ._llcrypt registered [ 1268.929735] Key type .llcrypt registered [ 1269.076815] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 1270.526078] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1270.557472] alg: No test for adler32 (adler32-zlib) [ 1271.756240] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 1272.190696] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 1273.855231] Key type lgssc registered [ 1275.272788] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1283.503785] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1283.535967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1285.092155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1289.373582] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1295.348292] LustreError: 48034:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1295.371577] LustreError: 48034:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1298.488823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1299.089558] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1303.749178] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1309.708737] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1312.947835] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1318.974126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1319.320833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1324.306401] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1329.169675] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1329.679766] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1334.835777] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1337.843237] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1350.031759] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2025-11-1416h09m58s [ 1351.386838] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1355.232956] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1355.236246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1355.236715] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1355.249580] Lustre: Skipped 2 previous similar messages [ 1355.261498] Lustre: Skipped 2 previous similar messages [ 1356.325471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1356.338458] Lustre: Skipped 2 previous similar messages [ 1357.574190] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.22@tcp (stopping) [ 1359.477082] LustreError: 50601:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1359.602832] Lustre: server umount lustre-MDT0000 complete [ 1362.409567] LustreError: 48747:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1362.455294] LustreError: 48747:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1363.191431] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1363.666751] LustreError: 48030:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1368.552672] LustreError: 48747:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1368.565805] LustreError: 48747:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1369.282415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1369.377542] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1369.603091] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1369.650320] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1370.617108] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1372.780872] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1375.900520] Lustre: DEBUG MARKER: Wait for df (36048) ... [ 1377.349815] Lustre: DEBUG MARKER: done [ 1380.836910] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1380.842108] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1380.852925] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1380.852939] Lustre: Skipped 1 previous similar message [ 1380.895533] Lustre: Skipped 1 previous similar message [ 1385.958160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1385.962198] Lustre: Skipped 2 previous similar messages [ 1386.623233] LustreError: 51657:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1386.632012] LustreError: 51657:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1386.763849] Lustre: server umount lustre-OST0000 complete [ 1391.073115] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1391.074506] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1397.217706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1397.221146] Lustre: Skipped 4 previous similar messages [ 1404.383386] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1404.462516] LustreError: 51860:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1404.473861] LustreError: 51860:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1404.560680] Lustre: server umount lustre-MDT0000 complete [ 1408.335693] LustreError: 48016:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763154658 with bad export cookie 15953598220740478126 [ 1408.339437] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1408.353032] LustreError: 48016:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1408.524880] LustreError: 52062:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1408.526937] LustreError: 52062:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1408.731740] Lustre: server umount lustre-MDT0001 complete [ 1419.327342] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 1422.492762] Key type lgssc unregistered [ 1422.789616] LNet: 52594:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1422.809976] LNetError: 52594:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1422.837500] LNet: Removed LNI 192.168.203.122@tcp [ 1423.732339] Key type .llcrypt unregistered [ 1423.734919] Key type ._llcrypt unregistered [ 1455.373273] Key type ._llcrypt registered [ 1455.384349] Key type .llcrypt registered [ 1455.529678] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 1470.089397] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 1471.074955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1471.150515] alg: No test for adler32 (adler32-zlib) [ 1472.181248] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 1472.391625] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 1474.047202] Key type lgssc registered [ 1475.074306] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1482.904310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1489.078672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1495.034304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1501.336896] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1507.660920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1507.723443] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1507.741832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1508.971952] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1509.018736] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1509.123092] Lustre: lustre-MDT0000: new disk, initializing [ 1509.239703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1509.271362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1512.897207] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1522.475991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1522.530843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1522.588145] Lustre: 56543:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1522.690795] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1522.697359] Lustre: Skipped 1 previous similar message [ 1522.793718] Lustre: lustre-MDT0001: new disk, initializing [ 1522.934609] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1523.004757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1523.021995] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1526.890393] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1533.813566] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1538.205709] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1545.403813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1545.509649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1545.731344] Lustre: lustre-OST0000: new disk, initializing [ 1545.753232] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1545.862974] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1552.514164] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1556.008893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1556.018448] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1556.132119] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1559.439795] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1562.920312] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1563.090533] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1565.926779] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1566.067524] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1571.301355] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1571.303528] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1571.313958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1571.347202] Lustre: Skipped 1 previous similar message [ 1574.106668] LustreError: 59484:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1574.193695] Lustre: server umount lustre-OST0000 complete [ 1579.489654] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1579.502624] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1579.516718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1579.519744] Lustre: Skipped 1 previous similar message [ 1581.543059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1581.544494] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1583.216989] LustreError: 59686:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1583.222252] LustreError: 59686:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1583.335390] Lustre: server umount lustre-MDT0000 complete [ 1586.303730] LustreError: 56534:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763154836 with bad export cookie 7035002131324255411 [ 1586.306341] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1586.312746] LustreError: 56534:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1586.450782] LustreError: 59888:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1586.453388] LustreError: 59888:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1586.547060] Lustre: server umount lustre-MDT0001 complete [ 1594.685910] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 16:14:03 (1763154843) [ 1600.916774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1601.434675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1604.839492] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1606.628331] LustreError: 60558:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1606.639314] LustreError: 60558:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1610.736314] LustreError: 60558:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1612.362302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1612.811034] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1616.403371] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1621.839546] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1625.364147] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1631.130963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1631.469169] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1636.944634] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1641.698579] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1652.734341] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2025-11-1416h15m01s [ 1653.983693] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1660.817665] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1663.377293] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1663.429293] LustreError: 60554:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1683.744496] Lustre: lustre-MDT0000: Client f0481f17-8b68-4882-827c-c036c9c114cf (at 192.168.203.22@tcp) reconnecting [ 1703.527117] LustreError: 60554:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1705.595646] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1707.971411] Lustre: DEBUG MARKER: done [ 1714.657981] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1714.663349] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1714.703361] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1714.708698] Lustre: Skipped 1 previous similar message [ 1717.736083] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1719.002355] LustreError: 63574:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1719.012174] LustreError: 63574:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1719.116776] Lustre: server umount lustre-OST0000 complete [ 1722.849409] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1722.851463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1722.877593] Lustre: Skipped 2 previous similar messages [ 1722.885773] Lustre: Skipped 3 previous similar messages [ 1728.552787] LustreError: 63775:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1728.557896] LustreError: 63775:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1728.646962] Lustre: server umount lustre-MDT0000 complete [ 1728.996225] LustreError: 61730:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1729.009565] LustreError: 61730:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1731.682076] LustreError: 60539:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763154982 with bad export cookie 7035002131324256293 [ 1731.688623] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1731.961949] Lustre: server umount lustre-MDT0001 complete [ 1742.413121] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 1745.739335] Key type lgssc unregistered [ 1746.055784] LNet: 64509:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1746.074087] LNetError: 64509:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1746.098790] LNet: Removed LNI 192.168.203.122@tcp [ 1746.997593] Key type .llcrypt unregistered [ 1746.999679] Key type ._llcrypt unregistered [ 1777.067454] Key type ._llcrypt registered [ 1777.069350] Key type .llcrypt registered [ 1777.176947] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 1789.212130] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 1790.267344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1790.308411] alg: No test for adler32 (adler32-zlib) [ 1791.351276] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 1791.538773] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 1793.191182] Key type lgssc registered [ 1794.098936] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1801.354646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1806.325585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1811.421749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1817.241823] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1823.081251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1823.112454] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1823.123524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1824.370321] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1824.417234] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1824.465755] Lustre: lustre-MDT0000: new disk, initializing [ 1824.613038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1824.638697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1827.529666] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1836.257251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1836.311402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1836.401669] Lustre: 68438:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1836.468814] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1836.476038] Lustre: Skipped 1 previous similar message [ 1836.538957] Lustre: lustre-MDT0001: new disk, initializing [ 1836.648053] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1836.706698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1836.717343] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1840.542805] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1847.652499] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1851.681933] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1857.638098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1857.692297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1857.856992] Lustre: lustre-OST0000: new disk, initializing [ 1857.862799] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1857.923084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1862.523187] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1864.262431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1864.281459] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1864.383594] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1869.251481] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1872.539418] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1872.762614] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1876.089944] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1876.302610] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1880.032051] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1880.043218] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1880.076093] LustreError: Skipped 1 previous similar message [ 1880.079228] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1880.103560] Lustre: Skipped 1 previous similar message [ 1885.156354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1885.160012] Lustre: Skipped 2 previous similar messages [ 1890.279973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1890.287079] Lustre: Skipped 1 previous similar message [ 1892.319151] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1892.363085] LustreError: 71377:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1892.442835] Lustre: server umount lustre-OST0000 complete [ 1897.952981] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1897.966794] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1897.997745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1900.522899] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1902.224111] LustreError: 71579:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1902.227274] LustreError: 71579:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1902.298901] Lustre: server umount lustre-MDT0000 complete [ 1905.905206] LustreError: 68429:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763155156 with bad export cookie 12916148752463223457 [ 1905.910375] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1905.917881] LustreError: 68429:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1906.063389] LustreError: 71781:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1906.073839] LustreError: 71781:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1906.290533] Lustre: server umount lustre-MDT0001 complete [ 1914.073633] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 16:19:22 (1763155162) [ 1915.633471] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1917.468045] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 16:19:26 (1763155166) [ 1926.740702] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 1929.754692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1930.096566] LustreError: 72829:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1930.111611] LustreError: 72829:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1930.140087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1930.972647] Lustre: Failing over lustre-MDT0000 [ 1930.977547] LustreError: 72858:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1930.983365] LustreError: 72858:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1930.988665] LustreError: 72858:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 1, retries 0, failed: rc = -5 [ 1931.111123] LustreError: 72909:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1931.114486] LustreError: 72909:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1931.264416] Lustre: server umount lustre-MDT0000 complete [ 1936.858157] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 16:19:45 (1763155185) [ 1942.396097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1942.808570] LustreError: 73479:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1942.864082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1946.069499] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1948.133660] LustreError: 73479:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1953.252805] LustreError: 73479:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1953.981700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1954.319959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1958.247986] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1963.987811] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1967.926883] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1975.065557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1975.465611] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1980.847894] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1985.441273] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1994.962855] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2003.428970] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2003.430906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2003.441035] Lustre: Skipped 2 previous similar messages [ 2003.438902] Lustre: Skipped 1 previous similar message [ 2007.405714] LustreError: 75825:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2007.419714] LustreError: 75825:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2007.554699] Lustre: server umount lustre-MDT0000 complete [ 2008.548535] LustreError: 74494:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2008.579327] LustreError: 74494:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2011.462308] LustreError: 73459:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763155261 with bad export cookie 12916148752463224619 [ 2011.467154] LustreError: 73459:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2011.469956] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2011.882168] Lustre: server umount lustre-MDT0001 complete [ 2015.184708] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2019.789634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2027.480452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2030.047180] Lustre: 66650:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763155264/real 1763155264] req@00000000f37ee81a x1848802061395712/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763155280 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2030.073189] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2030.092524] Lustre: Skipped 1 previous similar message [ 2033.183166] Lustre: 66650:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763155267/real 1763155267] req@0000000033ad79ff x1848802061395840/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763155283 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2035.167190] Lustre: 66652:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763155269/real 1763155269] req@00000000ae987fec x1848802061395968/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763155285 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2036.255804] LustreError: 66649:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000e9536957 x1848802061396736/t0(0) o250->MGC192.168.203.122@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2036.595519] LustreError: 75089:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2036.697199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2036.825894] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2037.663480] Lustre: 66653:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763155271/real 1763155271] req@000000001135f9b0 x1848802061396224/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763155287 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2040.581713] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2041.823761] Lustre: 66650:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763155276/real 1763155276] req@000000003e0ae325 x1848802061396352/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763155292 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2048.631046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2053.001742] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2054.128280] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2057.980233] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2059.300611] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2061.207884] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2069.012436] LustreError: 78284:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2069.025393] LustreError: 78284:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2069.104262] Lustre: server umount lustre-MDT0000 complete [ 2069.480184] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2069.487414] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2069.489148] LustreError: 77912:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2069.489157] LustreError: 77912:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2069.515359] Lustre: Skipped 2 previous similar messages [ 2072.561686] LustreError: 74198:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763155323 with bad export cookie 12916148752463226523 [ 2072.569302] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2072.894552] Lustre: server umount lustre-MDT0001 complete [ 2079.295531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2088.231644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2090.975215] Lustre: 66652:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763155325/real 1763155325] req@00000000fcd30084 x1848802061419136/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763155341 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2090.990261] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2098.707449] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2102.109763] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2105.829861] LustreError: 75089:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2105.855801] LustreError: 75089:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 2109.932740] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2110.048358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2110.350274] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2110.353453] Lustre: Skipped 2 previous similar messages [ 2114.462160] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2115.561413] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2119.710374] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2122.916114] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2130.777049] LustreError: 80742:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2130.781318] LustreError: 80742:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2130.883264] Lustre: server umount lustre-MDT0000 complete [ 2130.919330] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2130.924325] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2133.722892] LustreError: 73459:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763155384 with bad export cookie 12916148752463227895 [ 2133.729550] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2133.733639] LustreError: 73459:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2134.000906] Lustre: server umount lustre-MDT0001 complete [ 2139.691973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2142.885650] Lustre: DEBUG MARKER: files compared the same [ 2155.249775] Lustre: server umount lustre-OST0000 complete [ 2168.814626] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 2171.488373] Key type lgssc unregistered [ 2171.733948] LNet: 82374:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2171.749782] LNetError: 82374:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2171.767685] LNet: Removed LNI 192.168.203.122@tcp [ 2172.526897] Key type .llcrypt unregistered [ 2172.528505] Key type ._llcrypt unregistered [ 2194.923876] Key type ._llcrypt registered [ 2194.925610] Key type .llcrypt registered [ 2195.033925] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 2195.960924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2196.086031] alg: No test for adler32 (adler32-zlib) [ 2197.197305] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 2197.420276] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 2199.055171] Key type lgssc registered [ 2200.234795] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2208.186374] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2208.204633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2209.735482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2213.990932] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug +malloc all [ 2220.005693] LustreError: 83776:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2220.023642] LustreError: 83776:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2222.808280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2223.268383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2227.020566] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug +malloc all [ 2232.150901] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2235.209907] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2241.809708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2242.185192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2246.782804] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug +malloc all [ 2251.729449] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2254.538210] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2264.032962] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2264.035767] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2264.051131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2265.056269] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2265.059195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2265.067331] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2265.077669] Lustre: Skipped 1 previous similar message [ 2268.154278] LustreError: 86022:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2268.236293] Lustre: server umount lustre-OST0000 complete [ 2274.784397] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2274.790858] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2274.814892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2277.758391] LustreError: 86226:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2277.780955] LustreError: 86226:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2277.888949] Lustre: server umount lustre-MDT0000 complete [ 2280.415982] LustreError: 83775:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2280.432631] LustreError: 83775:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2281.078486] LustreError: 83755:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763155531 with bad export cookie 2702186854346634836 [ 2281.081726] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2281.092995] LustreError: 83755:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2281.207992] LustreError: 86427:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2281.211230] LustreError: 86427:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2281.347147] Lustre: server umount lustre-MDT0001 complete [ 2289.123354] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 2291.164303] Key type lgssc unregistered [ 2291.458513] LNet: 86958:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2291.470940] LNetError: 86958:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2291.490347] LNet: Removed LNI 192.168.203.122@tcp [ 2292.128415] Key type .llcrypt unregistered [ 2292.129771] Key type ._llcrypt unregistered [ 2312.057918] Key type ._llcrypt registered [ 2312.060683] Key type .llcrypt registered [ 2312.164052] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 2312.943701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2313.115816] alg: No test for adler32 (adler32-zlib) [ 2314.188950] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 2314.359446] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 2316.015119] Key type lgssc registered [ 2316.910483] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2323.923650] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2323.937128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2339.423336] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2344.543216] LustreError: 88307:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.122@tcp: failed processing log, type 4: rc = -110 [ 2371.167189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2377.129522] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2382.199671] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2389.533305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2389.638069] Lustre: *** cfs_fail_loc=706, val=0*** [ 2390.006864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2390.086447] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2393.745761] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2395.107544] LustreError: 88334:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2395.129432] LustreError: 88334:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2400.232254] LustreError: 88334:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2401.550296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2401.926774] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2405.284356] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2409.809490] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2413.188127] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2415.586332] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2415.605517] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2415.636537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2416.610760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2416.630436] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2421.109707] LustreError: 90632:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 2421.201780] Lustre: server umount lustre-OST0000 complete [ 2426.850104] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2426.851561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2426.854930] Lustre: Skipped 1 previous similar message [ 2426.865368] Lustre: Skipped 1 previous similar message [ 2430.504194] LustreError: 90833:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2430.507179] LustreError: 90833:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2430.696179] Lustre: server umount lustre-MDT0000 complete [ 2431.975949] LustreError: 89279:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2431.987844] LustreError: 89279:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2434.035021] LustreError: 88316:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763155684 with bad export cookie 5678490222886839035 [ 2434.058285] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2434.186454] LustreError: 91034:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2434.189550] LustreError: 91034:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2434.340466] Lustre: server umount lustre-MDT0001 complete [ 2443.682627] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 2446.976895] Key type lgssc unregistered [ 2447.278953] LNet: 91566:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2447.288405] LNetError: 91566:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2447.310846] LNet: Removed LNI 192.168.203.122@tcp [ 2448.365490] Key type .llcrypt unregistered [ 2448.367097] Key type ._llcrypt unregistered [ 2472.875872] Key type ._llcrypt registered [ 2472.885136] Key type .llcrypt registered [ 2473.034049] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 2474.157279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2474.177933] alg: No test for adler32 (adler32-zlib) [ 2475.272276] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 2475.475985] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 2477.200327] Key type lgssc registered [ 2478.236395] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2484.507849] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2484.543207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2488.848318] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2495.724897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2496.385796] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2499.871730] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2501.602793] LustreError: 93631:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2501.616852] LustreError: 93631:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2505.707543] LustreError: 93636:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2505.744027] LustreError: 93636:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2507.926196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2508.398535] LustreError: 93631:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2508.416575] LustreError: 93631:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2508.448432] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2513.888532] LustreError: 93635:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2513.927619] LustreError: 93635:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2515.730984] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2519.009812] LustreError: 93635:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2519.045770] LustreError: 93635:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2525.117598] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2525.130156] Lustre: MGS: Client a61c4a11-9d65-4d42-9122-0dc12207a452 (at 0@lo) reconnecting [ 2525.425891] LustreError: 93646:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2525.427557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2525.572829] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2528.805616] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2541.025588] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2541.032959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2541.056385] Lustre: Skipped 1 previous similar message [ 2541.077456] Lustre: Skipped 1 previous similar message [ 2545.841215] LustreError: 95597:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2545.980633] Lustre: server umount lustre-OST0000 complete [ 2551.264662] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2551.269704] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2551.294179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2551.303903] Lustre: Skipped 1 previous similar message [ 2551.312206] Lustre: Skipped 1 previous similar message [ 2555.259460] LustreError: 95799:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2555.270630] LustreError: 95799:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2555.306084] Lustre: server umount lustre-MDT0000 complete [ 2556.414515] LustreError: 93631:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2556.442202] LustreError: 93631:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2558.676498] LustreError: 96001:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2558.682707] LustreError: 96001:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2558.862285] Lustre: server umount lustre-MDT0001 complete [ 2561.994756] LustreError: 96202:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2561.997264] LustreError: 96202:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2562.068248] Lustre: server umount MGS complete [ 2571.211673] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 16:30:20 (1763155820) [ 2593.843681] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 2601.223672] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 2609.039286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2614.212216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2618.491681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2623.321926] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2628.763843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2628.829170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2631.782539] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2640.126195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2640.187777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2640.358405] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2640.409434] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2640.458569] Lustre: lustre-MDT0001: new disk, initializing [ 2640.515418] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2640.540623] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2640.546050] Lustre: 100716:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2640.554406] LustreError: 100716:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2644.062374] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2652.433536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2652.490861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2652.651446] Lustre: lustre-OST0000: new disk, initializing [ 2652.657494] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2652.736497] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2657.394958] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2663.457068] Lustre: 102008:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000a3e9035e x1848802777536128/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1763155929 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2664.573771] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2671.348936] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2671.361813] Lustre: MGS: Client a86572f5-40d8-46cd-bc3f-e73d5e900d72 (at 0@lo) reconnecting [ 2671.432510] Lustre: 100001:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2671.469789] LustreError: 100001:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 2671.526708] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2671.629113] Lustre: lustre-MDT0000: new disk, initializing [ 2671.815077] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2671.836720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2674.016997] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -11 [ 2676.219291] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2677.244878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2677.256758] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2678.141946] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2678.175732] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2702.825888] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2702.828172] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2702.832567] Lustre: Skipped 1 previous similar message [ 2702.837647] Lustre: Skipped 1 previous similar message [ 2705.812085] LustreError: 102951:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 2705.817253] LustreError: 102951:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2705.927917] Lustre: server umount lustre-OST0000 complete [ 2713.056855] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2713.056964] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2713.082813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2713.087690] Lustre: Skipped 1 previous similar message [ 2718.180200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2718.189954] Lustre: Skipped 1 previous similar message [ 2723.295261] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2723.434953] LustreError: 103155:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 2723.437724] LustreError: 103155:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2723.468739] Lustre: server umount lustre-MDT0000 complete [ 2732.822378] Lustre: server umount lustre-MDT0001 complete [ 2735.565235] Lustre: server umount MGS complete [ 2743.759718] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 16:33:12 (1763155992) [ 2757.804308] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 2760.324391] Key type lgssc unregistered [ 2760.697671] LNet: 104896:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2760.709558] LNetError: 104896:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2760.727993] LNet: Removed LNI 192.168.203.122@tcp [ 2761.504622] Key type .llcrypt unregistered [ 2761.506778] Key type ._llcrypt unregistered [ 2775.111346] Key type ._llcrypt registered [ 2775.113275] Key type .llcrypt registered [ 2775.229619] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 2776.268108] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2776.289951] alg: No test for adler32 (adler32-zlib) [ 2777.377717] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 2777.586733] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 2779.240368] Key type lgssc registered [ 2780.295629] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2787.753644] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2787.775663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2789.514644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2793.249583] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2799.077789] LustreError: 106094:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2799.111366] LustreError: 106094:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2801.385328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2801.764615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2806.231149] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2811.265376] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2814.125680] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2819.472024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2819.764507] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2825.347032] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2830.275850] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2834.162882] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2842.591959] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2842.599224] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2842.622493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2844.655099] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2844.655533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2844.671220] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2848.571420] LustreError: 108362:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2848.678260] Lustre: server umount lustre-OST0000 complete [ 2852.841573] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2852.846371] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2852.852240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2852.854544] Lustre: Skipped 1 previous similar message [ 2854.879925] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2854.880223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2854.893965] Lustre: Skipped 1 previous similar message [ 2860.003468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2860.010074] Lustre: Skipped 1 previous similar message [ 2866.143275] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2866.212480] LustreError: 108566:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2866.215136] LustreError: 108566:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2866.258917] Lustre: server umount lustre-MDT0000 complete [ 2869.404166] LustreError: 106075:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763156119 with bad export cookie 4640543707984790709 [ 2869.419342] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2869.573921] LustreError: 108768:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2869.599141] LustreError: 108768:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2869.776606] Lustre: server umount lustre-MDT0001 complete [ 2878.171982] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 2880.616875] Key type lgssc unregistered [ 2880.908336] LNet: 109299:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2880.912817] LNetError: 109299:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2881.965323] LNet: Removed LNI 192.168.203.122@tcp [ 2882.780573] Key type .llcrypt unregistered [ 2882.784687] Key type ._llcrypt unregistered [ 2895.555485] Key type ._llcrypt registered [ 2895.562583] Key type .llcrypt registered [ 2895.647238] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 2896.375886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2896.389979] alg: No test for adler32 (adler32-zlib) [ 2897.368281] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 2897.550284] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 2899.175163] Key type lgssc registered [ 2899.976850] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2904.803388] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2904.813667] LustreError: 110343:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 2904.822241] LustreError: 110340:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 2904.827086] LustreError: 110343:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4987 [ 2904.833070] LustreError: 110343:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2904.866650] LustreError: 110343:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2904.884432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2906.480130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2912.710333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2912.959406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2915.507663] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2919.041900] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing lsmod [ 2921.398803] LustreError: 111518:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 2921.406965] LustreError: 111519:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 2921.414193] LustreError: 111518:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4992 [ 2921.419490] LustreError: 111518:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2921.427273] LustreError: 111518:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2921.437159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2921.713396] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2928.107728] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2928.109567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2928.119626] Lustre: Skipped 2 previous similar messages [ 2928.121851] Lustre: Skipped 1 previous similar message [ 2933.218996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2933.231527] Lustre: Skipped 3 previous similar messages [ 2938.335432] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2938.341176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2938.349913] Lustre: Skipped 2 previous similar messages [ 2938.406824] LustreError: 111708:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2938.479669] Lustre: server umount lustre-MDT0000 complete [ 2940.920470] LustreError: 110362:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763156191 with bad export cookie 17729867518171450372 [ 2940.923640] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2940.930911] LustreError: 110362:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2941.028462] LustreError: 111914:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2941.033764] LustreError: 111914:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2941.140387] Lustre: server umount lustre-MDT0001 complete [ 2953.796319] LustreError: 112114:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2953.799234] LustreError: 112114:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2953.879459] Lustre: server umount lustre-OST0000 complete [ 2959.592683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2959.938668] LustreError: 112592:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2959.956514] LustreError: 112592:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2960.013611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2963.456241] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2965.474034] LustreError: 112591:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2965.495687] LustreError: 112591:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2969.572302] LustreError: 112592:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2970.582113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2970.892962] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2974.070508] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2977.589586] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2980.120226] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2985.461085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2985.709266] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2990.022477] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2994.220896] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3005.186956] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3015.652925] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3015.658034] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3015.691050] Lustre: Skipped 1 previous similar message [ 3015.703625] Lustre: Skipped 1 previous similar message [ 3020.228120] LustreError: 114856:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3020.230104] LustreError: 114856:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3020.360988] Lustre: server umount lustre-OST0000 complete [ 3025.888424] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3025.898557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3025.901521] Lustre: Skipped 1 previous similar message [ 3029.719588] LustreError: 115058:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3029.725788] LustreError: 115058:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3029.808942] Lustre: server umount lustre-MDT0000 complete [ 3031.008875] LustreError: 112591:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3032.832546] LustreError: 112571:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763156283 with bad export cookie 17729867518171451058 [ 3032.838890] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3032.848695] LustreError: 112571:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3033.173296] Lustre: server umount lustre-MDT0001 complete [ 3040.943589] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 3042.936121] Key type lgssc unregistered [ 3043.130587] LNet: 115791:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3043.140831] LNetError: 115791:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3043.155919] LNet: Removed LNI 192.168.203.122@tcp [ 3043.839120] Key type .llcrypt unregistered [ 3043.840689] Key type ._llcrypt unregistered [ 3061.839510] Key type ._llcrypt registered [ 3061.841047] Key type .llcrypt registered [ 3061.926600] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 3062.954887] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3062.968327] alg: No test for adler32 (adler32-zlib) [ 3063.990454] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 3064.213918] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 3065.855173] Key type lgssc registered [ 3066.810131] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3074.096063] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3074.111774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3075.553375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3078.940588] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3085.796361] LustreError: 117169:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3085.814046] LustreError: 117169:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3086.225690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3086.753920] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3090.347624] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3094.310327] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3097.009757] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3102.562477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3102.884852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3107.191894] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3111.556198] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3121.333082] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3125.364444] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3127.502853] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3131.874832] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3131.879121] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3131.892979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3136.060917] LustreError: 119532:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3136.165663] Lustre: server umount lustre-OST0000 complete [ 3142.113263] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3142.120903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3142.125460] Lustre: Skipped 2 previous similar messages [ 3142.132079] Lustre: Skipped 1 previous similar message [ 3145.625161] LustreError: 119741:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3145.627573] LustreError: 119741:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3145.701876] Lustre: server umount lustre-MDT0000 complete [ 3147.232900] LustreError: 118418:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3147.259566] LustreError: 118418:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3148.607042] LustreError: 117149:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763156399 with bad export cookie 4906166313516162793 [ 3148.608062] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3148.621192] LustreError: 117149:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3148.722769] LustreError: 119942:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3148.734480] LustreError: 119942:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3148.871673] Lustre: server umount lustre-MDT0001 complete [ 3156.787990] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 3159.103564] Key type lgssc unregistered [ 3159.357267] LNet: 120474:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3159.376273] LNetError: 120474:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3159.392723] LNet: Removed LNI 192.168.203.122@tcp [ 3160.038306] Key type .llcrypt unregistered [ 3160.039988] Key type ._llcrypt unregistered [ 3173.018804] Key type ._llcrypt registered [ 3173.020810] Key type .llcrypt registered [ 3173.098561] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 3173.656448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3173.664806] alg: No test for adler32 (adler32-zlib) [ 3174.671443] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 3174.824878] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 3176.471165] Key type lgssc registered [ 3177.405044] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3183.735436] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3183.771136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3185.116336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3188.434399] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3195.365865] LustreError: 121658:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3195.388143] LustreError: 121658:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3196.142615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3196.513536] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3199.585432] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3204.003904] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3206.833419] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3212.671839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3213.017317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3217.547817] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3221.724642] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3230.444779] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3240.935563] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3240.936197] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3240.938248] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3240.947332] Lustre: Skipped 1 previous similar message [ 3244.288210] LustreError: 123925:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3244.377885] Lustre: server umount lustre-OST0000 complete [ 3247.588784] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3247.593876] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3247.603421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3247.606529] Lustre: Skipped 1 previous similar message [ 3251.168631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3251.172198] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3253.044281] LustreError: 124127:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3253.050341] LustreError: 124127:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3253.116864] Lustre: server umount lustre-MDT0000 complete [ 3256.101460] LustreError: 121638:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763156506 with bad export cookie 9254829256395009763 [ 3256.107463] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3256.120965] LustreError: 121638:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3256.224027] LustreError: 124328:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3256.236267] LustreError: 124328:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3256.447839] Lustre: server umount lustre-MDT0001 complete [ 3263.184313] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 3265.031886] Key type lgssc unregistered [ 3265.266574] LNet: 124861:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3265.269952] LNetError: 124861:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3265.278599] LNet: Removed LNI 192.168.203.122@tcp [ 3265.905507] Key type .llcrypt unregistered [ 3265.907757] Key type ._llcrypt unregistered [ 3282.182283] Key type ._llcrypt registered [ 3282.183409] Key type .llcrypt registered [ 3282.242928] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 3282.714060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3282.762407] alg: No test for adler32 (adler32-zlib) [ 3283.708195] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 3283.852589] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 3285.479165] Key type lgssc registered [ 3286.188605] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3290.650272] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3290.665045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3291.938573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3294.144370] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3298.910291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3299.166312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3301.529144] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3304.973730] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3307.235373] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3311.420109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3311.694799] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3315.305745] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3316.844056] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3318.765868] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3328.458653] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3330.790602] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3330.793768] Lustre: Skipped 2 previous similar messages [ 3333.423218] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3333.426434] Lustre: Skipped 2 previous similar messages [ 3337.789764] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3337.802826] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3341.919200] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.22@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3341.924073] Lustre: Skipped 2 previous similar messages [ 3344.105942] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.22@tcp 192.168.0.[2,10]@tcp [ 3346.534319] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.22/24@tcp in log lustre-MDT0000 [ 3346.538177] Lustre: Skipped 2 previous similar messages [ 3352.093750] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.22/24@tcp [ 3352.097861] Lustre: Skipped 1 previous similar message [ 3358.182029] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3358.189527] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3358.203840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3360.227079] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3360.228762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3360.230109] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3360.249455] Lustre: Skipped 1 previous similar message [ 3361.973076] LustreError: 129883:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3362.054543] Lustre: server umount lustre-OST0000 complete [ 3365.350751] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3365.361677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3365.362391] Lustre: Skipped 1 previous similar message [ 3370.466165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3370.472027] Lustre: Skipped 2 previous similar messages [ 3370.646838] LustreError: 130085:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3370.653829] LustreError: 130085:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3370.732092] Lustre: server umount lustre-MDT0000 complete [ 3373.555660] LustreError: 126217:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763156624 with bad export cookie 31605765808192045 [ 3373.556669] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3373.560821] LustreError: 126217:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3373.671920] LustreError: 130287:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3373.675017] LustreError: 130287:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3373.841184] Lustre: server umount lustre-MDT0001 complete [ 3380.211341] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 3381.576878] Key type lgssc unregistered [ 3381.739129] LNet: 130819:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3381.742717] LNetError: 130819:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3381.756186] LNet: Removed LNI 192.168.203.122@tcp [ 3382.170459] Key type .llcrypt unregistered [ 3382.172786] Key type ._llcrypt unregistered [ 3399.199840] Key type ._llcrypt registered [ 3399.201471] Key type .llcrypt registered [ 3399.267515] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 3399.747598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3399.757360] alg: No test for adler32 (adler32-zlib) [ 3400.680533] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 3400.849868] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 3402.471183] Key type lgssc registered [ 3403.192237] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3408.626118] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3408.648477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3410.048877] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3410.061933] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.22/24@tcp [ 3410.079909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3413.157374] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3418.593415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3418.753334] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3418.766742] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.22/24@tcp [ 3418.790205] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3420.844702] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3423.785416] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3425.770757] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3429.391494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3429.568640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3432.575825] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3434.736316] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3435.685737] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3450.341738] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3450.353237] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3450.367108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3454.432696] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3454.435762] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3454.438171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3456.375366] LustreError: 134661:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3456.427649] Lustre: server umount lustre-OST0000 complete [ 3459.553804] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3459.556211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3459.560052] Lustre: Skipped 1 previous similar message [ 3459.566106] Lustre: Skipped 2 previous similar messages [ 3464.675632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3464.682841] Lustre: Skipped 1 previous similar message [ 3464.780161] LustreError: 134862:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3464.783270] LustreError: 134862:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3464.836078] Lustre: server umount lustre-MDT0000 complete [ 3466.805465] LustreError: 133964:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763156717 with bad export cookie 18422361494356870950 [ 3466.809682] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3466.824083] LustreError: 133964:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3466.900973] LustreError: 135063:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3466.904201] LustreError: 135063:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3466.987310] Lustre: server umount lustre-MDT0001 complete [ 3471.929851] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 3473.388579] Key type lgssc unregistered [ 3473.562159] LNet: 135594:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3473.567596] LNetError: 135594:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3474.600573] LNet: Removed LNI 192.168.203.122@tcp [ 3475.175278] Key type .llcrypt unregistered [ 3475.176994] Key type ._llcrypt unregistered [ 3489.737612] Key type ._llcrypt registered [ 3489.738979] Key type .llcrypt registered [ 3489.803927] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 3498.567298] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 3499.179306] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3499.200627] alg: No test for adler32 (adler32-zlib) [ 3500.162370] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 3500.344487] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 3502.007403] Key type lgssc registered [ 3502.710688] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3507.766778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3511.684482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3515.102838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3518.947283] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3522.753704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3522.780647] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3522.791300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3523.918741] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3523.936851] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3523.981703] Lustre: lustre-MDT0000: new disk, initializing [ 3524.026721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3524.039271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3526.018517] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3531.734286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3531.783666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3531.842839] Lustre: 138946:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3531.883938] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3531.886556] Lustre: Skipped 1 previous similar message [ 3531.941962] Lustre: lustre-MDT0001: new disk, initializing [ 3532.014931] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3532.037205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3532.041854] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3534.058918] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3538.318445] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3540.504811] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3544.722191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3544.772568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3544.923437] Lustre: lustre-OST0000: new disk, initializing [ 3544.930340] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3544.993472] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3548.353245] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3552.276835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3552.284528] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3552.360558] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3553.310648] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3555.766846] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3555.899043] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3558.572185] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3558.752487] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3562.466068] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3562.468049] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3562.469145] LustreError: Skipped 1 previous similar message [ 3562.469802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3562.476404] Lustre: Skipped 1 previous similar message [ 3562.487546] Lustre: Skipped 1 previous similar message [ 3567.585732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3567.590160] Lustre: Skipped 1 previous similar message [ 3572.705779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3572.709680] Lustre: Skipped 1 previous similar message [ 3574.239131] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3574.268154] LustreError: 141884:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3574.326452] Lustre: server umount lustre-OST0000 complete [ 3577.825205] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3577.827864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3577.832237] Lustre: Skipped 1 previous similar message [ 3582.498107] LustreError: 142087:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3582.501583] LustreError: 142087:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3582.566740] Lustre: server umount lustre-MDT0000 complete [ 3584.709727] LustreError: 138939:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763156835 with bad export cookie 75561953760964277 [ 3584.718098] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3584.720363] LustreError: 138939:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3584.806427] LustreError: 142288:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3584.809440] LustreError: 142288:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3584.944606] Lustre: server umount lustre-MDT0001 complete [ 3589.745115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3589.969485] LustreError: 142764:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3589.979746] LustreError: 142764:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3590.026529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3591.982158] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3593.056880] LustreError: 142764:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3596.585952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3596.790519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3598.877825] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3602.020128] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3604.038685] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3607.150598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3607.311266] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3609.756568] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3612.159118] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3614.170361] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3616.344888] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 3618.395803] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3622.657931] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 4 sec [ 3628.118599] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3628.153666] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3628.236348] Lustre: lustre-OST0001: new disk, initializing [ 3628.239288] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3628.291227] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3631.036133] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3633.685750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3633.692150] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3633.733753] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3635.092712] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3635.187879] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3636.943250] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3637.105363] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3638.703711] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9908c75da000.ost_server_uuid 50 [ 3639.406291] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9908c75da000.ost_server_uuid in FULL state after 0 sec [ 3643.876022] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3643.880298] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3643.887300] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3643.891389] Lustre: Skipped 2 previous similar messages [ 3648.243996] LustreError: 147054:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3648.246762] LustreError: 147054:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3648.271757] Lustre: server umount lustre-OST0001 complete [ 3648.997546] LustreError: 146339:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3649.005044] LustreError: 146339:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3651.554063] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3651.557587] LustreError: Skipped 1 previous similar message [ 3651.559872] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3651.564739] Lustre: Skipped 1 previous similar message [ 3652.579694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3652.586565] Lustre: Skipped 1 previous similar message [ 3654.115786] LustreError: 146339:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3654.123183] LustreError: 146339:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3659.236815] LustreError: 146339:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3659.244019] LustreError: 146339:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3664.351152] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3664.387045] LustreError: 147256:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3664.392921] LustreError: 147256:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3664.450046] Lustre: server umount lustre-OST0000 complete [ 3668.448515] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3668.453907] LustreError: Skipped 1 previous similar message [ 3668.456105] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3668.461682] Lustre: Skipped 1 previous similar message [ 3669.473617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3669.481371] Lustre: Skipped 8 previous similar messages [ 3681.247121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3681.294930] LustreError: 147458:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3681.298042] LustreError: 147458:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3681.337089] Lustre: server umount lustre-MDT0000 complete [ 3682.944251] LustreError: 145810:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763156933 with bad export cookie 75561953760965145 [ 3682.947740] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3682.949076] LustreError: 145810:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3683.097163] Lustre: server umount lustre-MDT0001 complete [ 3691.107994] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 3692.409318] Key type lgssc unregistered [ 3692.556748] LNet: 148649:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3692.560380] LNetError: 148649:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3692.570487] LNet: Removed LNI 192.168.203.122@tcp [ 3692.944055] Key type .llcrypt unregistered [ 3692.945598] Key type ._llcrypt unregistered [ 3708.530242] Key type ._llcrypt registered [ 3708.531897] Key type .llcrypt registered [ 3708.585378] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 3716.242865] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 3716.656642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3716.702911] alg: No test for adler32 (adler32-zlib) [ 3717.621639] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 3717.730755] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 3719.336662] Key type lgssc registered [ 3719.896081] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3724.319799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3727.283531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3729.929986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3732.904584] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3736.084825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3736.115494] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3736.123944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3737.242660] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3737.261565] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3737.311076] Lustre: lustre-MDT0000: new disk, initializing [ 3737.361401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3737.373295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3738.965995] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3744.564886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3744.613577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3744.662888] Lustre: 152572:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3744.696309] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3744.700507] Lustre: Skipped 1 previous similar message [ 3744.749762] Lustre: lustre-MDT0001: new disk, initializing [ 3744.811454] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3744.836373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3744.842770] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3746.729720] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3750.617764] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3752.283040] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3755.363197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3755.402776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3755.512836] Lustre: lustre-OST0000: new disk, initializing [ 3755.515280] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3755.552670] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3758.217807] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3759.249804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3759.255293] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3759.312250] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3762.571683] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3764.508689] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3764.609963] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3766.541540] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3766.649973] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3769.827548] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3769.828227] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3769.829120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3769.839485] Lustre: Skipped 1 previous similar message [ 3773.617827] LustreError: 155509:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3773.668199] Lustre: server umount lustre-OST0000 complete [ 3775.456604] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3775.461651] LustreError: Skipped 1 previous similar message [ 3775.464743] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3775.473095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3775.475829] Lustre: Skipped 1 previous similar message [ 3780.063930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3780.064306] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3781.453761] LustreError: 155710:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3781.456897] LustreError: 155710:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3781.494197] Lustre: server umount lustre-MDT0000 complete [ 3783.133523] LustreError: 152564:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763157033 with bad export cookie 13222388233076992983 [ 3783.137467] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3783.138881] LustreError: 152564:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3783.211662] LustreError: 155912:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3783.214936] LustreError: 155912:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3783.313959] Lustre: server umount lustre-MDT0001 complete [ 3786.911443] Lustre: DEBUG MARKER: == conf-sanity test 47a: server restart does not lose lru_resize ========================================================== 16:50:36 (1763157036) [ 3793.347544] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 3797.535979] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 3801.620789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3804.854902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3807.677290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3810.607310] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3813.385172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3813.416129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3813.506990] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3813.522699] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3813.564266] Lustre: lustre-MDT0000: new disk, initializing [ 3813.601650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3813.609520] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3815.110287] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3820.154441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3820.174675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3820.209077] Lustre: 158900:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3820.232256] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3820.234071] Lustre: Skipped 1 previous similar message [ 3820.270276] Lustre: lustre-MDT0001: new disk, initializing [ 3820.305282] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3820.318952] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3820.322105] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3821.845346] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3825.544346] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3826.969260] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3829.678773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3829.704375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3829.811975] Lustre: lustre-OST0000: new disk, initializing [ 3829.814587] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3829.846717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3832.170129] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3835.433604] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3837.433963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3837.437601] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3837.484759] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3842.706311] Lustre: Failing over lustre-OST0000 [ 3842.744377] LustreError: 161345:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3842.748798] LustreError: 161345:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3842.804736] Lustre: server umount lustre-OST0000 complete [ 3856.317569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3856.464046] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3856.470720] Lustre: Skipped 1 previous similar message [ 3856.475791] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3856.486039] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3857.110884] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3858.719699] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3860.433154] Lustre: Failing over lustre-MDT0000 [ 3860.506051] LustreError: 162317:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3860.508527] LustreError: 162317:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3860.541145] Lustre: server umount lustre-MDT0000 complete [ 3861.472351] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3861.476738] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3863.519206] Lustre: 150788:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763157098/real 1763157098] req@000000003665fd94 x1848804080939008/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1763157114 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3867.104061] LustreError: 158910:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3867.111545] LustreError: 158910:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 3868.767286] Lustre: 150786:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763157103/real 1763157103] req@00000000e3dc03a1 x1848804080940672/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1763157119 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3868.775105] Lustre: 150786:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3872.225098] LustreError: 158907:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3872.230725] LustreError: 158907:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3874.250399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3874.296147] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3874.300765] Lustre: 158902:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.203.122@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3874.378583] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 3874.430880] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3876.047225] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3877.583884] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3879.909298] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3879.920123] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 3879.948181] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3885.025697] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3885.026796] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3885.028986] LustreError: Skipped 1 previous similar message [ 3885.034481] Lustre: Skipped 2 previous similar messages [ 3885.035325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3885.040731] Lustre: Skipped 2 previous similar messages [ 3887.412189] LustreError: 163204:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3887.416662] LustreError: 163204:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3887.462840] Lustre: server umount lustre-OST0000 complete [ 3890.146912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3890.149693] Lustre: Skipped 1 previous similar message [ 3895.167364] Lustre: server umount lustre-MDT0000 complete [ 3895.264957] LustreError: 159696:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3895.272555] LustreError: 159696:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3896.729565] LustreError: 158891:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763157147 with bad export cookie 13222388233076995062 [ 3896.733029] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3896.734618] LustreError: 158891:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3896.887124] Lustre: server umount lustre-MDT0001 complete [ 3901.118306] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 3902.321516] Key type lgssc unregistered [ 3902.454578] LNet: 164138:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3902.458244] LNetError: 164138:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3902.466555] LNet: Removed LNI 192.168.203.122@tcp [ 3902.823855] Key type .llcrypt unregistered [ 3902.825602] Key type ._llcrypt unregistered [ 3923.983587] Key type ._llcrypt registered [ 3923.984805] Key type .llcrypt registered [ 3924.026473] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 3924.468538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3924.483175] alg: No test for adler32 (adler32-zlib) [ 3925.362296] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 3925.462979] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 3927.047177] Key type lgssc registered [ 3927.469539] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3930.199432] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3930.207066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3931.368452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3932.810410] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3936.140462] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3938.186356] LustreError: 165662:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3938.190724] LustreError: 165662:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3938.194820] LustreError: 165662:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 3938.263874] LustreError: 166184:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3938.322371] Lustre: server umount lustre-MDT0000 complete [ 3943.690540] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 3947.015019] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 3950.449432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3952.660780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3954.782559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3957.326365] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3960.077583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3960.107222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3960.203521] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3960.216185] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3960.254664] Lustre: lustre-MDT0000: new disk, initializing [ 3960.281252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3960.287769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3961.848272] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3966.079778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3966.103726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3966.136084] Lustre: 168825:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3966.165464] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3966.167442] Lustre: Skipped 1 previous similar message [ 3966.204296] Lustre: lustre-MDT0001: new disk, initializing [ 3966.240844] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3966.256372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3966.259460] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3967.658777] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3971.057169] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3972.387405] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3974.894388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3974.927729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3975.036331] Lustre: lustre-OST0000: new disk, initializing [ 3975.039118] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3975.072533] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3977.098018] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3977.151830] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3977.156234] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3977.205853] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3980.335468] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3982.236766] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 3982.818940] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 16:53:52 (1763157232) [ 4079.072128] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4079.075572] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4079.081438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4079.583766] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4079.584053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4079.589751] Lustre: Skipped 1 previous similar message [ 4083.580731] LustreError: 172339:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4083.582769] LustreError: 172339:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4083.619253] Lustre: server umount lustre-MDT0000 complete [ 4084.703550] LustreError: 168836:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4084.710628] LustreError: 168836:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4084.930509] LustreError: 168815:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763157335 with bad export cookie 5301424592750985048 [ 4084.931546] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4084.934905] LustreError: 168815:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4089.823512] LustreError: 168831:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4089.823553] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4089.836028] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4089.838377] Lustre: Skipped 1 previous similar message [ 4091.070267] LustreError: 172541:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4091.072857] LustreError: 172541:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4091.114703] Lustre: server umount lustre-MDT0001 complete [ 4098.544313] LustreError: 172742:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4098.546701] LustreError: 172742:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4098.585205] Lustre: server umount lustre-OST0000 complete [ 4101.748480] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 4105.076660] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 4108.899357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4111.256901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4113.426868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4115.835749] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4118.687301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4118.723740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4118.838597] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4118.858738] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4118.903283] Lustre: lustre-MDT0000: new disk, initializing [ 4118.961341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4118.981409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4120.648074] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4125.303981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4125.332707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4125.376385] Lustre: 175076:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4125.408116] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4125.410864] Lustre: Skipped 1 previous similar message [ 4125.454761] Lustre: lustre-MDT0001: new disk, initializing [ 4125.514866] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4125.519849] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4127.194646] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4130.773667] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4132.180567] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4135.132624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4135.169451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4135.282951] Lustre: lustre-OST0000: new disk, initializing [ 4135.285227] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4135.323942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4135.327441] Lustre: Skipped 1 previous similar message [ 4137.599922] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4140.029144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4140.033927] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4140.072593] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4141.029945] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4142.560935] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4142.650605] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4144.103340] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4144.189384] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4145.122187] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4145.122250] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4145.130518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4145.132348] Lustre: Skipped 1 previous similar message [ 4146.926100] LustreError: 178016:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4146.927964] LustreError: 178016:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4146.971599] Lustre: server umount lustre-OST0000 complete [ 4150.241767] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4150.249515] Lustre: Skipped 1 previous similar message [ 4150.253433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4150.255404] Lustre: Skipped 1 previous similar message [ 4154.399892] Lustre: server umount lustre-MDT0000 complete [ 4155.360504] LustreError: 175087:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4155.367886] LustreError: 175087:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4155.863171] LustreError: 175857:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763157406 with bad export cookie 5301424592751143864 [ 4155.866159] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4155.869850] LustreError: 175857:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4155.940556] LustreError: 178418:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4155.943116] LustreError: 178418:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 4155.998740] Lustre: server umount lustre-MDT0001 complete [ 4159.089242] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 16:56:48 (1763157408) [ 4165.053290] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 4168.809459] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 4172.898835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4175.884424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4178.641538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4181.402458] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4184.656252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4184.682599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4184.793252] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4184.810649] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4184.855209] Lustre: lustre-MDT0000: new disk, initializing [ 4184.897853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4184.909745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4186.685636] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4192.067252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4192.095794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4192.145846] Lustre: 181405:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4192.273387] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4193.878982] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4197.452125] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4198.797430] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4201.443630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4201.470877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4203.638224] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4204.800921] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4204.850436] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4207.020180] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4210.145206] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4210.145298] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4210.147922] LustreError: Skipped 2 previous similar messages [ 4210.152925] Lustre: Skipped 1 previous similar message [ 4210.155832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4210.159916] Lustre: Skipped 1 previous similar message [ 4215.342152] LustreError: 183950:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4215.344734] LustreError: 183950:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4215.400191] Lustre: server umount lustre-OST0000 complete [ 4217.824259] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4223.103347] Lustre: server umount lustre-MDT0000 complete [ 4224.591418] LustreError: 182189:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763157475 with bad export cookie 5301424592751144746 [ 4224.593744] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4224.596995] LustreError: 182189:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4224.720739] Lustre: server umount lustre-MDT0001 complete [ 4227.991971] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 16:57:57 (1763157477) [ 4234.283183] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 4238.003626] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 4242.041332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4244.542239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4246.946347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4249.920574] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4253.486863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4253.519951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4253.641187] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4253.659055] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4253.665093] Lustre: Skipped 3 previous similar messages [ 4253.714623] Lustre: lustre-MDT0000: new disk, initializing [ 4253.717921] Lustre: Skipped 2 previous similar messages [ 4253.779331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4253.783071] Lustre: Skipped 2 previous similar messages [ 4253.794229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4253.797379] Lustre: Skipped 2 previous similar messages [ 4255.643107] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4261.147902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4261.187265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4261.245138] Lustre: 187340:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4261.395713] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4263.248914] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4267.051980] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4268.611519] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4271.302696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4271.330766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4272.807427] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4273.637030] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4277.083518] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4283.360863] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4283.360972] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4283.364268] LustreError: Skipped 1 previous similar message [ 4283.365370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4283.369221] Lustre: Skipped 3 previous similar messages [ 4283.375646] Lustre: Skipped 6 previous similar messages [ 4285.357204] LustreError: 189886:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4285.360211] LustreError: 189886:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4285.404862] Lustre: server umount lustre-OST0000 complete [ 4301.279135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4301.358433] Lustre: server umount lustre-MDT0000 complete [ 4302.816963] LustreError: 188132:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4302.823360] LustreError: 188132:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4302.900406] LustreError: 187331:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763157553 with bad export cookie 5301424592751145831 [ 4302.903703] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4302.907416] LustreError: 187331:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4303.043451] Lustre: server umount lustre-MDT0001 complete [ 4307.391670] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 4308.651882] Key type lgssc unregistered [ 4308.792552] LNet: 190823:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4308.795738] LNetError: 190823:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4308.809926] LNet: Removed LNI 192.168.203.122@tcp [ 4309.206260] Key type .llcrypt unregistered [ 4309.207761] Key type ._llcrypt unregistered [ 4319.751694] Key type ._llcrypt registered [ 4319.753316] Key type .llcrypt registered [ 4319.797964] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 4320.266250] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4320.279954] alg: No test for adler32 (adler32-zlib) [ 4321.155217] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 4321.255571] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 4322.847206] Key type lgssc registered [ 4323.315776] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4326.292598] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4326.302634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4327.487073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4329.037400] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4332.714888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4332.892903] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4334.472309] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4336.924356] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4338.645238] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4341.758971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4341.936833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4344.410665] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4346.708235] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4352.166840] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4367.842659] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4367.844816] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4367.846516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4367.851468] Lustre: Skipped 1 previous similar message [ 4370.094413] LustreError: 194462:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4370.141922] Lustre: server umount lustre-OST0000 complete [ 4372.961756] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4372.962528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4372.966087] Lustre: Skipped 1 previous similar message [ 4372.968155] Lustre: Skipped 1 previous similar message [ 4377.728915] LustreError: 194663:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4377.731235] LustreError: 194663:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4377.760285] Lustre: server umount lustre-MDT0000 complete [ 4378.080436] LustreError: 192196:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4378.086638] LustreError: 192196:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4379.116524] LustreError: 192177:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763157629 with bad export cookie 3588073415301417452 [ 4379.119627] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4379.120894] LustreError: 192177:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4379.195899] LustreError: 194867:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4379.197857] LustreError: 194867:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4379.250148] Lustre: server umount lustre-MDT0001 complete [ 4383.204159] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 4384.408733] Key type lgssc unregistered [ 4384.571591] LNet: 195397:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4384.575547] LNetError: 195397:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4384.582440] LNet: Removed LNI 192.168.203.122@tcp [ 4384.972750] Key type .llcrypt unregistered [ 4384.974728] Key type ._llcrypt unregistered [ 4394.863857] Key type ._llcrypt registered [ 4394.865633] Key type .llcrypt registered [ 4394.909329] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 4395.268042] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4395.310111] alg: No test for adler32 (adler32-zlib) [ 4396.183294] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 4396.275446] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 4397.863177] Key type lgssc registered [ 4398.292995] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4401.322084] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4401.330376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4402.491649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4403.915337] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4407.106624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4407.282917] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4408.844427] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4411.064246] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4412.557191] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4415.399575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4415.568857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4417.970737] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4420.264479] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4426.854862] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4432.085971] Lustre: lustre-OST0000: Not available for connect from 192.168.203.22@tcp (stopping) [ 4432.356186] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4432.357187] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4432.364524] Lustre: Skipped 1 previous similar message [ 4435.314340] LustreError: 199028:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4435.354660] Lustre: server umount lustre-OST0000 complete [ 4437.303218] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9908d0096000.ost_server_uuid 50 [ 4437.894490] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 4448.223746] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4448.226031] LustreError: Skipped 1 previous similar message [ 4448.227398] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4448.231417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4448.233467] Lustre: Skipped 2 previous similar messages [ 4452.321133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4452.322166] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4454.078311] LustreError: 199376:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4454.081224] LustreError: 199376:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4454.114700] Lustre: server umount lustre-MDT0000 complete [ 4455.471125] LustreError: 196754:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763157705 with bad export cookie 14269195605126567452 [ 4455.474552] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4455.475305] LustreError: 196754:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4455.542055] LustreError: 199577:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4455.544621] LustreError: 199577:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4455.607128] Lustre: server umount lustre-MDT0001 complete [ 4458.694283] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 17:01:48 (1763157708) [ 4461.141174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4461.318204] LustreError: 200248:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4461.326770] LustreError: 200248:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4461.347486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4462.743309] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4465.595536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4465.725072] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4467.074951] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4468.860279] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4470.028430] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4472.289428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4474.352459] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4476.240921] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4478.537843] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4478.561926] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4478.623101] Lustre: lustre-OST0001: new disk, initializing [ 4478.625672] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4478.663367] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4478.666589] Lustre: Skipped 1 previous similar message [ 4479.242972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4479.247299] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4479.290555] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4480.756897] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4484.132417] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4487.139071] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4487.139774] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4487.142847] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4487.145227] Lustre: Skipped 1 previous similar message [ 4491.181093] LustreError: 203450:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4491.183023] LustreError: 203450:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4491.201289] Lustre: server umount lustre-OST0000 complete [ 4492.258194] LustreError: 202411:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4492.264784] LustreError: 202411:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4493.465122] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4493.544045] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4494.544465] LustreError: 201845:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4494.937645] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4495.037047] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4497.376835] LustreError: 201845:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4497.382610] LustreError: 201845:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4502.499800] LustreError: 202411:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4502.505942] LustreError: 202411:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4507.619976] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4507.623065] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4507.623798] Lustre: Skipped 1 previous similar message [ 4507.625683] Lustre: Skipped 1 previous similar message [ 4511.151668] LustreError: 204152:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 4511.154413] LustreError: 204152:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4511.202658] Lustre: server umount lustre-OST0001 complete [ 4516.832431] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4516.835455] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4516.843845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4516.846392] Lustre: Skipped 1 previous similar message [ 4527.071136] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4527.109827] LustreError: 204354:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4527.112304] LustreError: 204354:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4527.146797] Lustre: server umount lustre-MDT0000 complete [ 4528.096389] LustreError: 200248:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4528.104184] LustreError: 200248:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4528.484283] LustreError: 202751:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763157778 with bad export cookie 14269195605126568509 [ 4528.487924] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4528.488632] LustreError: 202751:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4528.622734] Lustre: server umount lustre-MDT0001 complete [ 4530.662901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4532.217867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4533.796516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4535.320697] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4537.888589] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 17:03:07 (1763157787) [ 4540.456772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4540.583290] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4540.600438] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4540.688435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4542.192529] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4545.207055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4545.245234] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4545.260440] Lustre: 206047:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4546.673390] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4548.493033] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4549.762399] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4552.170208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4552.240213] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4554.306284] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4555.408335] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4556.306825] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4558.778810] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4558.804393] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4560.798090] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4562.641347] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4563.494152] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 4565.985593] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4565.985844] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4565.989430] LustreError: Skipped 1 previous similar message [ 4565.994698] Lustre: Skipped 1 previous similar message [ 4565.997200] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4566.000961] Lustre: Skipped 5 previous similar messages [ 4569.710208] LustreError: 209240:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4569.712573] LustreError: 209240:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4569.731831] Lustre: server umount lustre-OST0000 complete [ 4573.391530] LustreError: 207674:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4583.906152] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4583.910941] LustreError: Skipped 1 previous similar message [ 4586.580830] Lustre: server umount lustre-OST0001 complete [ 4599.264499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4599.266957] Lustre: Skipped 10 previous similar messages [ 4602.335230] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4602.373147] LustreError: 209645:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4602.375734] LustreError: 209645:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4602.410556] Lustre: server umount lustre-MDT0000 complete [ 4603.834202] LustreError: 206037:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763157854 with bad export cookie 14269195605126569643 [ 4603.837959] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4603.839054] LustreError: 206037:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4603.977546] Lustre: server umount lustre-MDT0001 complete [ 4606.142311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4607.543719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4608.875901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4610.210284] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4612.751514] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 17:04:22 (1763157862) [ 4618.426938] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 4621.733772] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 4625.306469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4627.467582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4629.608425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4631.944831] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4634.509322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4634.547809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4634.651194] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4634.666045] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4634.708993] Lustre: lustre-MDT0000: new disk, initializing [ 4634.741410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4634.744375] Lustre: Skipped 3 previous similar messages [ 4634.751216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4636.313537] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4640.787846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4640.815962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4640.854189] Lustre: 213664:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4640.879833] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4640.882800] Lustre: Skipped 1 previous similar message [ 4640.919763] Lustre: lustre-MDT0001: new disk, initializing [ 4640.966759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4640.970853] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4642.372854] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4645.655171] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4646.962316] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4649.420758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4649.444099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4649.543560] Lustre: lustre-OST0000: new disk, initializing [ 4649.546766] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4651.006396] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4651.009987] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4651.047175] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4651.619188] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4654.896119] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4656.286176] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4656.368139] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4657.740420] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4657.829567] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4661.218187] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4661.218399] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4661.223053] Lustre: Skipped 5 previous similar messages [ 4661.227014] LustreError: Skipped 1 previous similar message [ 4664.657462] Lustre: server umount lustre-OST0000 complete [ 4666.337120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4666.338884] Lustre: Skipped 3 previous similar messages [ 4680.159122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4680.193773] LustreError: 216805:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4680.196270] LustreError: 216805:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 4680.227638] Lustre: server umount lustre-MDT0000 complete [ 4680.672063] LustreError: 214451:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4680.677937] LustreError: 214451:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 4681.595660] LustreError: 215411:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763157932 with bad export cookie 14269195605126571400 [ 4681.600041] LustreError: 215411:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4681.600091] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4681.728094] Lustre: server umount lustre-MDT0001 complete [ 4684.917850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4686.496685] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4689.560614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4691.134159] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4693.239464] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4694.572655] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4697.138295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4699.414188] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4701.412252] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4702.929554] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4710.186132] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec [ 4711.631230] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4711.728063] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4714.977266] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4718.546617] Lustre: server umount lustre-OST0000 complete [ 4720.630745] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4722.222345] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4746.543664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4746.666247] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4746.669678] Lustre: Skipped 5 previous similar messages [ 4748.703754] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4750.723925] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4751.312165] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4754.388626] Lustre: server umount lustre-OST0000 complete [ 4756.447949] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4756.452954] Lustre: Skipped 4 previous similar messages [ 4757.974963] Lustre: server umount lustre-MDT0000 complete [ 4759.308410] LustreError: 217463:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763158009 with bad export cookie 14269195605126572275 [ 4759.311653] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4759.311916] LustreError: 217463:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4759.431659] Lustre: server umount lustre-MDT0001 complete [ 4762.560813] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 17:06:52 (1763158012) [ 4765.048399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4765.227591] LustreError: 222984:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4765.235516] LustreError: 222984:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4766.715880] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4769.762433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4771.345605] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4773.198088] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4774.398972] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4776.888156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4779.122428] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4781.231848] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4782.763750] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4791.066483] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 4792.627639] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4792.724186] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4795.330791] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4795.367289] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4795.438933] Lustre: lustre-OST0001: new disk, initializing [ 4795.441201] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4797.645922] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4801.000094] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4802.581193] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4803.076263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4803.079455] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4803.113222] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4803.701969] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 4805.207355] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4805.296259] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4808.166372] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4808.169179] LustreError: Skipped 2 previous similar messages [ 4808.169447] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4808.173188] Lustre: Skipped 8 previous similar messages [ 4812.080185] LustreError: 227244:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 4812.083245] LustreError: 227244:0:(obd_class.h:479:obd_check_dev()) Skipped 35 previous similar messages [ 4812.108474] Lustre: server umount lustre-OST0001 complete [ 4814.379095] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4814.465731] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4815.888452] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4815.979276] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4839.599442] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4841.576300] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4843.581895] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4844.152603] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4850.933619] Lustre: server umount lustre-OST0001 complete [ 4869.589681] Lustre: server umount lustre-OST0000 complete [ 4877.100646] Lustre: server umount lustre-MDT0000 complete [ 4878.532372] LustreError: 222965:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763158129 with bad export cookie 14269195605126573206 [ 4878.535318] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4878.536820] LustreError: 222965:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4884.736395] Lustre: server umount lustre-MDT0001 complete [ 4886.861347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4888.239851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4889.524368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4890.907940] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4893.269399] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 17:09:02 (1763158142) [ 4895.640083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4895.749155] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4895.760884] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4895.832344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4895.834785] Lustre: Skipped 5 previous similar messages [ 4897.122316] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4899.996179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4900.030804] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4900.041447] Lustre: 231017:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4901.515744] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4903.380311] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4904.638444] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4907.054694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4907.120090] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4909.158782] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4911.002753] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4919.930701] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4919.956204] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4921.866683] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4923.619321] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4925.142312] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 4926.392668] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4926.473759] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4927.786221] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4927.867200] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4929.022905] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9908c5838000.ost_server_uuid 50 [ 4929.540052] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9908c5838000.ost_server_uuid in FULL state after 0 sec [ 4929.860836] Lustre: Permanently deactivating lustre-OST0001 [ 4929.866366] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 4930.612815] Lustre: Permanently reactivating lustre-OST0001 [ 4936.163502] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4936.166944] Lustre: Skipped 7 previous similar messages [ 4937.595723] Lustre: server umount lustre-OST0001 complete [ 4941.283197] LustreError: 233205:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4941.287080] LustreError: 233205:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 23 previous similar messages [ 4944.978723] Lustre: server umount lustre-OST0000 complete [ 4946.400922] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4946.402847] LustreError: Skipped 3 previous similar messages [ 4952.340966] Lustre: server umount lustre-MDT0000 complete [ 4953.664185] LustreError: 231007:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763158204 with bad export cookie 14269195605126574466 [ 4953.667620] LustreError: 231007:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4953.668347] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4953.797215] Lustre: server umount lustre-MDT0001 complete [ 4955.740410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4957.043768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4958.354223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4959.637939] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4961.885048] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 17:10:11 (1763158211) [ 4962.233131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4964.534821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4964.652551] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4964.666320] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4964.672103] Lustre: Skipped 5 previous similar messages [ 4966.231179] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4969.269270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4969.306353] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4969.316592] Lustre: 237151:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4970.767583] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4972.606363] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4973.842403] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4976.284049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4976.311996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4976.383752] Lustre: Permanently deactivating lustre-OST0000 [ 4976.388145] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 4978.421861] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4980.300798] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4982.620373] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4982.871668] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4982.923971] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 4984.647725] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4986.504940] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4987.148827] Lustre: Permanently reactivating lustre-OST0000 [ 4988.345320] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 4988.347691] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4988.353584] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4988.370833] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 4988.378717] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5002.366456] Lustre: server umount lustre-OST0001 complete [ 5009.682705] Lustre: server umount lustre-OST0000 complete [ 5017.173706] Lustre: server umount lustre-MDT0000 complete [ 5018.424265] LustreError: 237144:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763158268 with bad export cookie 14269195605126576104 [ 5018.427972] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5018.428612] LustreError: 237144:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5018.544919] Lustre: server umount lustre-MDT0001 complete [ 5021.812866] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 5022.762527] Key type lgssc unregistered [ 5022.879274] LNet: 241540:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5022.881285] LNetError: 241540:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5022.887301] LNet: Removed LNI 192.168.203.122@tcp [ 5023.181923] Key type .llcrypt unregistered [ 5023.183604] Key type ._llcrypt unregistered [ 5031.156914] Key type ._llcrypt registered [ 5031.158520] Key type .llcrypt registered [ 5031.196952] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 5031.537719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5031.558528] alg: No test for adler32 (adler32-zlib) [ 5032.394252] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 5032.463857] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 5034.031144] Key type lgssc registered [ 5034.379247] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5035.469563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5037.727684] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5037.733458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5038.891165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5040.165404] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5042.992216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5043.013877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5043.049379] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5043.059323] Lustre: 242957:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5043.062638] Lustre: Permanently deactivating lustre-MDT0001 [ 5043.067668] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5043.173126] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5044.467158] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5045.228654] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5045.231385] LustreError: 242999:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5045.234454] LustreError: 242999:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5045.239623] LustreError: 242999:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 1, failed: rc = -108 [ 5046.301155] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5047.498312] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5049.899647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5050.010604] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5052.060074] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5053.092782] LustreError: 244588:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5053.095063] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 5053.099108] LustreError: 244588:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5053.942966] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5056.370175] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5056.450444] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5058.455254] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5060.391520] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5061.103906] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 5064.169238] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 5064.169299] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 5067.400142] Lustre: Permanently reactivating lustre-MDT0001 [ 5067.407032] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 5067.409686] Lustre: Skipped 2 previous similar messages [ 5069.716367] Lustre: lustre-MDT0001: Client 25dc32e4-9162-475b-8a9b-74491c599130 (at 192.168.203.22@tcp) reconnecting [ 5071.787724] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5072.393451] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5074.133242] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 5074.221659] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 5075.557317] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 5075.645750] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 5076.542669] Lustre: Permanently deactivating lustre-MDT0001 [ 5076.547566] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5076.549862] Lustre: Skipped 2 previous similar messages [ 5082.620320] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5087.203613] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 5087.207871] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5087.213382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5090.784361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5090.786524] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5090.791646] Lustre: Skipped 2 previous similar messages [ 5093.442481] LustreError: 247032:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 5093.471544] Lustre: server umount lustre-MDT0000 complete [ 5094.746100] LustreError: 242947:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763158345 with bad export cookie 18016923256117096875 [ 5094.749749] LustreError: 242947:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5094.749914] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5094.822018] LustreError: 247234:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 5094.825113] LustreError: 247234:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5094.876993] Lustre: server umount lustre-MDT0001 complete [ 5102.189077] LustreError: 247435:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 5102.191616] LustreError: 247435:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5102.208995] Lustre: server umount lustre-OST0000 complete [ 5106.140220] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 17:12:35 (1763158355) [ 5109.400290] LustreError: 248441:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5109.402179] LustreError: 248441:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5109.436827] Lustre: server umount lustre-OST0001 complete [ 5111.306705] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 5114.029724] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 5117.064213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5119.013065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5120.927202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5122.913657] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5125.107858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5125.125514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5125.203923] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5125.206199] Lustre: Skipped 2 previous similar messages [ 5125.213240] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5125.243823] Lustre: lustre-MDT0000: new disk, initializing [ 5125.264910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5125.272291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5126.428746] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5130.626994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5130.648651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5130.684569] Lustre: 250622:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5130.710713] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5130.713332] Lustre: Skipped 1 previous similar message [ 5130.758639] Lustre: lustre-MDT0001: new disk, initializing [ 5130.800740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5130.804829] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5132.106499] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5135.234644] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5136.465814] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5138.787606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5138.811078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5138.904630] Lustre: lustre-OST0000: new disk, initializing [ 5138.906688] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5138.934614] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5138.938243] Lustre: Skipped 1 previous similar message [ 5140.956982] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5144.111657] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5145.080916] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5145.083795] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5145.120806] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5149.933041] LustreError: 252374:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5154.309841] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5154.334843] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5154.396805] Lustre: lustre-OST0001: new disk, initializing [ 5154.398746] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5154.484750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5154.488362] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5154.503496] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000400 [ 5156.397725] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5159.551899] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5160.023134] LustreError: 252374:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5161.038588] LustreError: 252374:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5161.181723] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 5171.135130] LustreError: 252374:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5175.267325] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5175.267845] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5175.269695] LustreError: Skipped 1 previous similar message [ 5175.270374] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5175.274828] Lustre: Skipped 1 previous similar message [ 5175.276981] Lustre: Skipped 3 previous similar messages [ 5177.899216] LustreError: 254119:0:(obd_class.h:479:obd_check_dev()) Device 26 not setup [ 5177.902209] LustreError: 254119:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5177.922910] Lustre: server umount lustre-OST0001 complete [ 5180.387344] LustreError: 252974:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5180.391632] LustreError: 252974:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5185.507476] LustreError: 252974:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5185.511132] LustreError: 252974:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5190.626167] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5190.626827] LustreError: 252974:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5190.627655] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5190.627659] Lustre: Skipped 2 previous similar messages [ 5190.630317] Lustre: Skipped 1 previous similar message [ 5190.638617] LustreError: 252974:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5195.747388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5195.749131] Lustre: Skipped 2 previous similar messages [ 5195.749498] LustreError: 252349:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5195.756218] LustreError: 252349:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5195.886089] LustreError: 254342:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5195.888137] LustreError: 254342:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5195.932389] Lustre: server umount lustre-OST0000 complete [ 5197.279900] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5197.282231] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5203.499129] Lustre: server umount lustre-MDT0000 complete [ 5204.805652] LustreError: 250614:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763158455 with bad export cookie 18016923256117099689 [ 5204.808372] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5204.809068] LustreError: 250614:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5219.295111] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5219.326987] LustreError: 254744:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5219.329434] LustreError: 254744:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 5219.390596] Lustre: server umount lustre-MDT0001 complete [ 5222.838744] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 5223.820632] Key type lgssc unregistered [ 5223.953392] LNet: 255277:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5223.955694] LNetError: 255277:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5223.966345] LNet: Removed LNI 192.168.203.122@tcp [ 5224.284274] Key type .llcrypt unregistered [ 5224.285760] Key type ._llcrypt unregistered [ 5235.306144] Key type ._llcrypt registered [ 5235.307153] Key type .llcrypt registered [ 5235.348058] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 5241.441627] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 5241.798174] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5241.849543] alg: No test for adler32 (adler32-zlib) [ 5242.703626] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 5242.787644] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 5244.375148] Key type lgssc registered [ 5244.725352] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5247.744777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5249.909039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5251.969387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5254.177297] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5256.552416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5256.571550] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5256.576851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5257.656707] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5257.667973] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5257.705568] Lustre: lustre-MDT0000: new disk, initializing [ 5257.732970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5257.740030] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5259.078308] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5263.277645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5263.303354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5263.334893] Lustre: 259198:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5263.357873] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5263.359592] Lustre: Skipped 1 previous similar message [ 5263.393364] Lustre: lustre-MDT0001: new disk, initializing [ 5263.425085] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5263.439394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5263.443840] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5264.822900] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5267.942648] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5269.192966] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5271.441242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5271.463671] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5271.558077] Lustre: lustre-OST0000: new disk, initializing [ 5271.559862] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5271.588210] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5273.015034] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5273.018103] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5273.055083] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5273.465214] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5276.571989] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5277.997409] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5278.080366] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5279.509741] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5279.602462] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5282.286088] LustreError: 262135:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5282.318085] Lustre: server umount lustre-OST0000 complete [ 5283.807956] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5283.810138] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5283.813777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5288.417155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5288.417702] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5293.537187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5293.539522] Lustre: Skipped 2 previous similar messages [ 5293.540171] LustreError: 262336:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5293.542401] LustreError: 262336:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5293.565801] Lustre: server umount lustre-MDT0000 complete [ 5294.864274] LustreError: 259188:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763158545 with bad export cookie 7349405771412101362 [ 5294.867843] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5294.869812] LustreError: 259188:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5305.315164] LustreError: 262538:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5305.316970] LustreError: 262538:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5305.364475] Lustre: server umount lustre-MDT0001 complete [ 5308.377590] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 17:15:58 (1763158558) [ 5310.675558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5310.846682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5312.175399] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5315.149653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5315.286131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5316.651896] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5318.489239] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5319.705848] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5322.124571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5324.178958] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5326.131371] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5376.479867] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5376.482523] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5376.486814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5377.506599] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5381.487165] LustreError: 265555:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5381.489056] LustreError: 265555:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5381.533168] Lustre: server umount lustre-OST0000 complete [ 5383.029325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 5387.154530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5387.259619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5387.261690] Lustre: Skipped 1 previous similar message [ 5389.145244] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5390.970649] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5397.536783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5397.540403] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5402.656321] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5402.660456] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5428.194250] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5428.194816] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5428.197270] LustreError: Skipped 1 previous similar message [ 5428.203579] Lustre: Skipped 1 previous similar message [ 5428.205638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5428.209278] Lustre: Skipped 3 previous similar messages [ 5430.574117] LustreError: 267220:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5430.576027] LustreError: 267220:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5430.627214] Lustre: server umount lustre-OST0000 complete [ 5433.313079] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5433.313140] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5433.318559] Lustre: Skipped 1 previous similar message [ 5438.097117] Lustre: server umount lustre-MDT0000 complete [ 5438.432059] LustreError: 263211:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5438.437490] LustreError: 263211:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5439.313357] LustreError: 263192:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763158689 with bad export cookie 7349405771412102237 [ 5439.317103] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5439.317383] LustreError: 263192:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5439.393741] LustreError: 267625:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5439.395670] LustreError: 267625:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 5439.448164] Lustre: server umount lustre-MDT0001 complete [ 5443.033553] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 5444.100617] Key type lgssc unregistered [ 5444.225417] LNet: 268155:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5444.228630] LNetError: 268155:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5444.241299] LNet: Removed LNI 192.168.203.122@tcp [ 5444.550804] Key type .llcrypt unregistered [ 5444.552736] Key type ._llcrypt unregistered [ 5453.871664] Key type ._llcrypt registered [ 5453.872871] Key type .llcrypt registered [ 5453.909199] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 5454.223914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5454.260086] alg: No test for adler32 (adler32-zlib) [ 5455.120035] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 5455.223504] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 5456.807224] Key type lgssc registered [ 5457.171175] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5459.789812] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5459.797525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5460.958338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5462.244342] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5465.152215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5465.279917] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5466.601132] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5468.415428] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5469.706687] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5472.022395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5472.163846] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5474.087324] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5475.877171] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5487.526240] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 5500.895894] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5500.897952] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5500.901572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5502.944662] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5502.946882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5502.946993] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5504.620107] LustreError: 272570:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5504.658721] Lustre: server umount lustre-OST0000 complete [ 5506.527958] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5506.530708] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5506.534659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5506.536321] Lustre: Skipped 1 previous similar message [ 5513.184296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5513.186197] Lustre: Skipped 3 previous similar messages [ 5518.303959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5518.305866] Lustre: Skipped 1 previous similar message [ 5520.351092] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5520.387348] LustreError: 272772:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5520.389572] LustreError: 272772:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5520.413187] Lustre: server umount lustre-MDT0000 complete [ 5521.663211] LustreError: 269511:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763158772 with bad export cookie 1488062362962475236 [ 5521.666395] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5521.669363] LustreError: 269511:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5521.726395] LustreError: 272974:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5521.729310] LustreError: 272974:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5521.779701] Lustre: server umount lustre-MDT0001 complete [ 5525.446548] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 5526.552108] Key type lgssc unregistered [ 5526.683407] LNet: 273505:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5526.688543] LNetError: 273505:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5526.700367] LNet: Removed LNI 192.168.203.122@tcp [ 5527.010543] Key type .llcrypt unregistered [ 5527.011573] Key type ._llcrypt unregistered [ 5534.084142] Key type ._llcrypt registered [ 5534.085201] Key type .llcrypt registered [ 5534.119285] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 5534.441582] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5534.485326] alg: No test for adler32 (adler32-zlib) [ 5535.347571] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 5535.423721] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 5536.999201] Key type lgssc registered [ 5537.381574] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5539.986968] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5539.992496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5541.153058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5542.468611] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5545.367505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5545.488488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5546.763209] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5548.554069] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5549.754681] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5552.047431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5552.159050] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5554.045750] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5555.843780] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5567.526633] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5575.647812] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5575.651762] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5575.657815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5577.698268] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5577.698631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5577.699133] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5577.706247] Lustre: Skipped 1 previous similar message [ 5580.911089] LustreError: 277098:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5580.949259] Lustre: server umount lustre-OST0000 complete [ 5582.816617] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5582.822458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5582.824553] Lustre: Skipped 1 previous similar message [ 5587.936089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5587.938390] Lustre: Skipped 1 previous similar message [ 5588.283919] LustreError: 277299:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5588.287081] LustreError: 277299:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5588.315955] Lustre: server umount lustre-MDT0000 complete [ 5589.673956] LustreError: 274665:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763158840 with bad export cookie 14660279293318146345 [ 5589.677554] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5589.678663] LustreError: 274665:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5589.744199] LustreError: 277502:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5589.746503] LustreError: 277502:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5589.796672] Lustre: server umount lustre-MDT0001 complete [ 5593.370220] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 5594.427369] Key type lgssc unregistered [ 5594.555345] LNet: 278032:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5594.557895] LNetError: 278032:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5594.565321] LNet: Removed LNI 192.168.203.122@tcp [ 5594.860629] Key type .llcrypt unregistered [ 5594.862153] Key type ._llcrypt unregistered [ 5604.282960] Key type ._llcrypt registered [ 5604.284147] Key type .llcrypt registered [ 5604.329980] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 5604.662375] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5604.697088] alg: No test for adler32 (adler32-zlib) [ 5605.558333] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 5605.641573] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 5607.223171] Key type lgssc registered [ 5607.601975] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5610.213130] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5610.220335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5611.404049] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5612.813800] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5615.808710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5615.942981] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5617.224298] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5619.047248] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5620.274915] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5622.597357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5622.707767] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5624.757335] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5626.623506] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5636.006907] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5646.308102] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5646.309562] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5646.310716] LustreError: Skipped 1 previous similar message [ 5646.311486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5646.316478] Lustre: Skipped 1 previous similar message [ 5646.320386] Lustre: Skipped 1 previous similar message [ 5647.020106] LustreError: 282448:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5647.055980] Lustre: server umount lustre-OST0000 complete [ 5650.364564] LustreError: 282649:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5650.366526] LustreError: 282649:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5650.390799] Lustre: server umount lustre-MDT0000 complete [ 5651.424438] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5651.428619] LustreError: 279425:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5651.433097] LustreError: 279425:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5651.606144] LustreError: 281023:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763158902 with bad export cookie 11540901036972260134 [ 5651.609295] LustreError: 281023:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5651.609374] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5651.671366] LustreError: 282852:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5651.674324] LustreError: 282852:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5651.721412] Lustre: server umount lustre-MDT0001 complete [ 5655.414882] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 5656.476230] Key type lgssc unregistered [ 5656.605610] LNet: 283383:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5656.607806] LNetError: 283383:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5656.617310] LNet: Removed LNI 192.168.203.122@tcp [ 5656.902511] Key type .llcrypt unregistered [ 5656.903881] Key type ._llcrypt unregistered [ 5664.110391] Key type ._llcrypt registered [ 5664.111418] Key type .llcrypt registered [ 5664.145702] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 5664.470544] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5664.530844] alg: No test for adler32 (adler32-zlib) [ 5665.383773] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 5665.475894] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 5667.063128] Key type lgssc registered [ 5667.422330] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5669.988301] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5669.995974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5671.175305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5672.591102] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5675.544299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5675.673897] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5677.055837] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5678.934830] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5680.282017] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5682.666987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5682.785644] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5684.743254] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5686.600650] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5696.103299] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 5701.601617] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5701.603540] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5701.604840] LustreError: Skipped 1 previous similar message [ 5701.612151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5701.616031] Lustre: Skipped 1 previous similar message [ 5702.957215] LustreError: 287010:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5702.998413] Lustre: server umount lustre-OST0000 complete [ 5706.496031] LustreError: 287213:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5706.498875] LustreError: 287213:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5706.529649] Lustre: server umount lustre-MDT0000 complete [ 5706.720687] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5706.720985] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5706.724703] LustreError: 284586:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5706.728790] Lustre: Skipped 2 previous similar messages [ 5706.737527] LustreError: 284586:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5707.873181] LustreError: 286924:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763158958 with bad export cookie 1585790506098684107 [ 5707.876095] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5707.878316] LustreError: 286924:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5707.948912] LustreError: 287413:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5707.951990] LustreError: 287413:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5708.014807] Lustre: server umount lustre-MDT0001 complete [ 5711.904672] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 5712.884927] Key type lgssc unregistered [ 5713.000362] LNet: 287945:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5713.002791] LNetError: 287945:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5713.009307] LNet: Removed LNI 192.168.203.122@tcp [ 5713.286322] Key type .llcrypt unregistered [ 5713.287342] Key type ._llcrypt unregistered [ 5717.168367] Key type ._llcrypt registered [ 5717.169979] Key type .llcrypt registered [ 5717.218512] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 5722.543541] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 5728.673930] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 5728.986935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5729.030395] alg: No test for adler32 (adler32-zlib) [ 5729.876985] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 5729.959328] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 5731.551185] Key type lgssc registered [ 5731.944081] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5734.739804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5736.804780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5738.741426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5740.790914] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5743.046505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5743.061157] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5743.065737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5744.146743] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5744.156536] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5744.195085] Lustre: lustre-MDT0000: new disk, initializing [ 5744.218986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5744.227926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5745.390811] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5749.279460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5749.301187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5749.333228] Lustre: 291427:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5749.355027] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5749.356822] Lustre: Skipped 1 previous similar message [ 5749.393091] Lustre: lustre-MDT0001: new disk, initializing [ 5749.424543] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5749.435424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5749.438661] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5750.591626] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5753.782625] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5755.158213] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5757.507361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5757.534756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5757.633460] Lustre: lustre-OST0000: new disk, initializing [ 5757.636025] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5757.664767] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5759.721670] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5762.974698] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5763.573811] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5763.576807] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5763.613199] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5764.296161] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5764.384629] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5765.712377] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5765.801972] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5768.673414] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5768.674604] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5768.676449] LustreError: Skipped 1 previous similar message [ 5768.677166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5768.680081] Lustre: Skipped 1 previous similar message [ 5768.684719] Lustre: Skipped 1 previous similar message [ 5772.524185] LustreError: 294365:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5772.559659] Lustre: server umount lustre-OST0000 complete [ 5773.792430] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5773.793109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5773.797025] Lustre: Skipped 1 previous similar message [ 5773.802276] Lustre: Skipped 1 previous similar message [ 5778.912078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5778.913967] Lustre: Skipped 1 previous similar message [ 5779.836727] LustreError: 294566:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5779.839497] LustreError: 294566:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5779.869652] Lustre: server umount lustre-MDT0000 complete [ 5781.093425] LustreError: 292208:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763159031 with bad export cookie 1325219262423415138 [ 5781.096925] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5781.099604] LustreError: 292208:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5781.164169] LustreError: 294767:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5781.165986] LustreError: 294767:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5781.216348] Lustre: server umount lustre-MDT0001 complete [ 5784.045551] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 17:23:53 (1763159033) [ 5786.278491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5786.451632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5787.743726] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5790.577806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5790.703488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5791.962682] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5793.750545] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5794.998264] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5797.301181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5799.416260] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5801.214090] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5815.778112] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5815.779922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5815.785058] Lustre: Skipped 1 previous similar message [ 5815.790559] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5820.783130] LustreError: 297705:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5820.785084] LustreError: 297705:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5820.836782] Lustre: server umount lustre-OST0000 complete [ 5824.993106] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 5824.995418] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5825.001974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5825.003726] Lustre: Skipped 1 previous similar message [ 5828.289649] LustreError: 297907:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5828.292315] LustreError: 297907:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5828.318970] Lustre: server umount lustre-MDT0000 complete [ 5829.566234] LustreError: 295416:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763159080 with bad export cookie 1325219262423416006 [ 5829.568920] LustreError: 296159:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 5829.569386] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5829.569811] LustreError: 295416:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5829.689927] Lustre: server umount lustre-MDT0001 complete [ 5833.395082] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 5834.517089] Key type lgssc unregistered [ 5834.649375] LNet: 298641:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5834.652673] LNetError: 298641:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5834.663354] LNet: Removed LNI 192.168.203.122@tcp [ 5834.997340] Key type .llcrypt unregistered [ 5834.999301] Key type ._llcrypt unregistered [ 5843.471558] Key type ._llcrypt registered [ 5843.473084] Key type .llcrypt registered [ 5843.510360] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 5849.822722] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 5850.290288] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5850.299236] alg: No test for adler32 (adler32-zlib) [ 5851.149487] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 5851.245382] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 5852.823159] Key type lgssc registered [ 5853.174612] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5855.938690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5858.058490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5860.078636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5862.227658] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5863.973461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5865.802479] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5868.105456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5868.127861] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5868.132892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5869.216976] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5869.229479] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5869.269478] Lustre: lustre-MDT0000: new disk, initializing [ 5869.294877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5869.301728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5870.569838] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5874.808027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5874.832302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5874.866273] Lustre: 302422:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5874.892022] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5874.893814] Lustre: Skipped 1 previous similar message [ 5874.929103] Lustre: lustre-MDT0001: new disk, initializing [ 5874.964040] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5874.977492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5874.981844] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5876.281512] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5879.383944] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5880.624944] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5882.919423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5882.948332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5883.061974] Lustre: lustre-OST2710: new disk, initializing [ 5883.063752] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 5883.098213] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 5885.058705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 5885.061866] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 5885.103775] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 5885.115926] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5888.259752] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 5890.546035] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5890.573722] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5890.632177] Lustre: lustre-OST03e8: new disk, initializing [ 5890.634091] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 5890.666108] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 5892.628487] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5893.316755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 5893.319843] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 5893.355205] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 5894.483618] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 5897.142567] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 5897.224973] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5898.595305] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 5898.679398] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5900.012648] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 5900.088133] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5901.454260] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 5901.543948] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5903.843148] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5903.844830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5903.848146] Lustre: Skipped 3 previous similar messages [ 5903.853553] Lustre: Skipped 3 previous similar messages [ 5908.736769] LustreError: 306760:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5908.763668] Lustre: server umount lustre-MDT0000 complete [ 5910.032638] LustreError: 303201:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763159160 with bad export cookie 9788037235542539507 [ 5910.036682] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5910.038299] LustreError: 303201:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5910.109023] LustreError: 306961:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5910.111226] LustreError: 306961:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5910.167948] Lustre: server umount lustre-MDT0001 complete [ 5921.270098] LustreError: 307161:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5921.273306] LustreError: 307161:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5921.295555] Lustre: server umount lustre-OST2710 complete [ 5932.850971] LustreError: 307363:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5932.853626] LustreError: 307363:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5932.908604] Lustre: server umount lustre-OST03e8 complete [ 5938.967060] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 5942.160078] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 5945.535353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5947.598367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5949.696898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5951.893954] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5954.203848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5954.225141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5954.311093] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5954.321227] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5954.359564] Lustre: lustre-MDT0000: new disk, initializing [ 5954.383164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5954.389910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5955.727635] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5960.107457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5960.130150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5960.160777] Lustre: 310153:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5960.253174] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5961.533207] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5964.934428] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5966.166067] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5968.352583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5968.377043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5968.461904] Lustre: lustre-OST0000: new disk, initializing [ 5968.463290] Lustre: Skipped 1 previous similar message [ 5968.464907] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5968.466828] Lustre: Skipped 2 previous similar messages [ 5968.492817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5968.494905] Lustre: Skipped 1 previous similar message [ 5970.325071] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5971.255979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5971.259817] Lustre: Skipped 1 previous similar message [ 5971.262136] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5971.287242] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5973.475359] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5974.834572] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5974.911987] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5976.229505] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5976.304985] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5981.664725] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5981.666229] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5981.667668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5981.670023] Lustre: Skipped 1 previous similar message [ 5986.785830] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5986.788163] Lustre: Skipped 2 previous similar messages [ 5991.391124] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5991.410106] LustreError: 313094:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5991.413897] LustreError: 313094:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5991.450046] Lustre: server umount lustre-OST0000 complete [ 5995.999963] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5996.003114] LustreError: Skipped 1 previous similar message [ 5996.005011] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5996.011571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6002.144076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6002.145841] Lustre: Skipped 3 previous similar messages [ 6007.263308] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6007.302243] LustreError: 313296:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6007.304010] LustreError: 313296:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6007.335774] Lustre: server umount lustre-MDT0000 complete [ 6008.615912] LustreError: 310145:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763159259 with bad export cookie 9788037235542541173 [ 6008.619805] LustreError: 310145:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6008.620399] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6008.750074] Lustre: server umount lustre-MDT0001 complete [ 6011.627297] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 17:27:41 (1763159261) [ 6012.172819] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 6012.798469] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 17:27:42 (1763159262) [ 6013.792249] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 6016.472773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6017.794800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6019.097105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6020.407365] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6020.781767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6023.093331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6023.178360] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6023.186843] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6023.246178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6024.572656] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6027.624807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6027.660774] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6027.670915] Lustre: 315625:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6029.223442] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6031.083387] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6032.325201] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6034.567303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6034.590744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6034.656948] Lustre: Denying initial registration attempt from nid 192.168.203.122@tcp, specified as failover [ 6034.661393] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 6034.664290] LustreError: 317238:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -99 [ 6034.666226] LustreError: 317238:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 6034.668782] LustreError: 317238:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 6034.690633] Lustre: server umount lustre-OST0000 complete [ 6034.692100] LustreError: 317238:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 6037.983728] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6037.986335] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6037.990357] Lustre: Skipped 1 previous similar message [ 6037.992234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6037.994442] Lustre: Skipped 2 previous similar messages [ 6051.295166] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6051.344593] LustreError: 317605:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 6051.347155] LustreError: 317605:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6051.371593] Lustre: server umount lustre-MDT0000 complete [ 6052.647141] LustreError: 315615:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763159303 with bad export cookie 9788037235542542041 [ 6052.650293] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6052.653069] LustreError: 315615:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6052.759091] Lustre: server umount lustre-MDT0001 complete [ 6056.462379] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 6057.592307] Key type lgssc unregistered [ 6057.739312] LNet: 318340:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6057.742026] LNetError: 318340:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6057.749385] LNet: Removed LNI 192.168.203.122@tcp [ 6058.098147] Key type .llcrypt unregistered [ 6058.099324] Key type ._llcrypt unregistered [ 6061.851350] Key type ._llcrypt registered [ 6061.852332] Key type .llcrypt registered [ 6061.898700] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 6062.274391] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6062.318128] alg: No test for adler32 (adler32-zlib) [ 6063.187265] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 6063.280477] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 6064.871208] Key type lgssc registered [ 6065.307158] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6067.792493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6069.158808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6070.530965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6071.946366] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6072.320128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6079.156114] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 6082.103208] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6082.109647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6083.206488] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6083.216053] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6083.279846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6084.671166] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6087.676877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6087.708416] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6087.718305] Lustre: 321073:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6087.815140] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6089.141598] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6090.992847] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6092.328063] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6094.694567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6094.722861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6094.793693] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6094.867048] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6096.948619] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6098.876319] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6103.009637] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6103.010174] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6103.011904] LustreError: Skipped 1 previous similar message [ 6103.016616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6103.019434] Lustre: Skipped 1 previous similar message [ 6105.644064] LustreError: 323359:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6105.679647] Lustre: server umount lustre-OST0000 complete [ 6108.128805] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6108.129508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6108.132499] Lustre: Skipped 2 previous similar messages [ 6108.136444] Lustre: Skipped 1 previous similar message [ 6113.085753] LustreError: 323560:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6113.088715] LustreError: 323560:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6113.115966] Lustre: server umount lustre-MDT0000 complete [ 6114.382354] LustreError: 321063:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763159364 with bad export cookie 13320515146935743646 [ 6114.385574] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6114.386203] LustreError: 321063:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6114.461488] LustreError: 323762:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6114.464478] LustreError: 323762:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6114.528954] Lustre: server umount lustre-MDT0001 complete [ 6118.355420] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 6119.358142] Key type lgssc unregistered [ 6119.481495] LNet: 324292:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6119.483920] LNetError: 324292:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6119.493353] LNet: Removed LNI 192.168.203.122@tcp [ 6119.837312] Key type .llcrypt unregistered [ 6119.838395] Key type ._llcrypt unregistered [ 6129.490080] Key type ._llcrypt registered [ 6129.491393] Key type .llcrypt registered [ 6129.532725] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 6129.989753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6130.019982] alg: No test for adler32 (adler32-zlib) [ 6130.896968] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 6130.984806] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 6132.567195] Key type lgssc registered [ 6132.943974] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6135.619167] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6135.626138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6136.797804] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6138.228964] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6141.249863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6141.385375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6142.788325] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6144.757656] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6146.068793] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6148.457503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6148.592596] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6150.645407] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6152.693770] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6167.009354] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6167.009403] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6167.010571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6167.020012] Lustre: Skipped 1 previous similar message [ 6168.278323] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.22@tcp (stopping) [ 6168.281755] Lustre: Skipped 2 previous similar messages [ 6171.385590] LustreError: 327935:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6171.413593] Lustre: server umount lustre-MDT0000 complete [ 6172.130653] LustreError: 325661:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6172.137182] LustreError: 325661:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6172.717333] LustreError: 325646:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763159423 with bad export cookie 680797809824569765 [ 6172.722829] LustreError: 325646:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6172.723181] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6173.390726] LustreError: 327606:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6173.399355] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.22@tcp (stopping) [ 6177.248127] LustreError: 325661:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6177.248447] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6177.260085] Lustre: Skipped 1 previous similar message [ 6177.262617] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6182.367906] LustreError: 325663:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6182.368165] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6182.372938] LustreError: 325663:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6182.375047] Lustre: Skipped 1 previous similar message [ 6186.975099] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6187.009842] LustreError: 328138:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6187.012341] LustreError: 328138:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6187.067844] Lustre: server umount lustre-MDT0001 complete [ 6189.306489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6192.172190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6197.727867] LustreError: 324976:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b0605601 x1848806611116672/t0(0) o250->MGC192.168.203.122@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6197.865303] LustreError: 327273:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6197.874438] LustreError: 327273:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6197.893934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6197.914329] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 6199.270576] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6202.127994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6202.240332] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6203.474680] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6205.241801] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6213.607497] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6213.992482] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 8 sec [ 6215.264822] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6215.819560] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6217.696281] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6217.698729] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6217.703319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6217.705659] Lustre: Skipped 2 previous similar messages [ 6218.721122] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6223.215093] LustreError: 330442:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6223.217120] LustreError: 330442:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6223.254396] Lustre: server umount lustre-OST0000 complete [ 6227.935911] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6227.939687] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6227.945578] Lustre: Skipped 1 previous similar message [ 6230.652755] LustreError: 330643:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6230.654897] LustreError: 330643:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6230.679268] Lustre: server umount lustre-MDT0000 complete [ 6231.912391] LustreError: 325646:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763159482 with bad export cookie 680797809824576884 [ 6231.913952] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6231.916218] LustreError: 325646:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6232.048423] Lustre: server umount lustre-MDT0001 complete [ 6236.106842] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 6237.251243] Key type lgssc unregistered [ 6237.377452] LNet: 331376:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6237.380259] LNetError: 331376:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6237.390471] LNet: Removed LNI 192.168.203.122@tcp [ 6237.720711] Key type .llcrypt unregistered [ 6237.721888] Key type ._llcrypt unregistered [ 6247.148223] Key type ._llcrypt registered [ 6247.149213] Key type .llcrypt registered [ 6247.186580] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 6247.476219] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6247.492434] alg: No test for adler32 (adler32-zlib) [ 6248.348361] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 6248.434966] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 6250.015162] Key type lgssc registered [ 6250.410647] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6253.095237] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6253.101791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6254.269978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6255.642583] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6258.613159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6258.740137] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6260.079599] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6261.862541] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6263.095667] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6268.898051] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6268.898303] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6268.898758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6268.902397] Lustre: Skipped 1 previous similar message [ 6269.921716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6269.923914] Lustre: Skipped 2 previous similar messages [ 6270.272670] LustreError: 334138:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6270.305098] Lustre: server umount lustre-MDT0000 complete [ 6271.582657] LustreError: 332731:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763159522 with bad export cookie 11925488338103482653 [ 6271.585934] LustreError: 332731:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6271.586827] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6271.654418] LustreError: 334341:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6271.656279] LustreError: 334341:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6271.708968] Lustre: server umount lustre-MDT0001 complete [ 6276.677882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6276.779444] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6276.790397] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6276.866244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6278.193476] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6281.098480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6281.131298] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6281.143982] Lustre: 335109:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6281.236238] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6282.639313] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6284.454500] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6285.675477] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6288.330645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6288.402599] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 6288.406741] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 6288.409864] LustreError: 336770:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 6288.412224] LustreError: 336770:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 6288.414232] LustreError: 336770:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 6288.437166] Lustre: server umount lustre-OST0000 complete [ 6288.438473] LustreError: 336770:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 6291.286301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6291.357462] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6291.424734] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6293.283434] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6295.078897] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6297.776586] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6297.804190] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6297.831681] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6297.864818] Lustre: lustre-OST0001: new disk, initializing [ 6297.866758] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6298.903755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6298.910463] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6298.933716] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 6298.975198] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6299.937519] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6301.722290] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6304.228064] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6304.228826] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6304.235837] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6304.238770] LustreError: Skipped 1 previous similar message [ 6308.783081] LustreError: 338885:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 6308.784865] LustreError: 338885:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6308.805702] Lustre: server umount lustre-OST0001 complete [ 6309.348866] LustreError: 337236:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6309.357719] LustreError: 337236:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6314.469749] LustreError: 337236:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6314.470926] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6314.471536] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6314.471541] Lustre: Skipped 1 previous similar message [ 6314.472103] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6314.472106] Lustre: Skipped 1 previous similar message [ 6314.474651] LustreError: 337236:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6314.478707] LustreError: Skipped 1 previous similar message [ 6316.141488] LustreError: 339086:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6316.144954] LustreError: 339086:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6316.184332] Lustre: server umount lustre-OST0000 complete [ 6319.584902] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6319.585733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6319.590128] Lustre: Skipped 2 previous similar messages [ 6319.593553] Lustre: Skipped 2 previous similar messages [ 6323.589674] LustreError: 339289:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 6323.593072] LustreError: 339289:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6323.637939] Lustre: server umount lustre-MDT0000 complete [ 6324.704486] LustreError: 335935:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6324.709168] LustreError: 335935:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6324.894421] LustreError: 335101:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763159575 with bad export cookie 11925488338103483192 [ 6324.898203] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6324.900988] LustreError: 335101:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6325.019032] Lustre: server umount lustre-MDT0001 complete [ 6328.580405] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 6329.652515] Key type lgssc unregistered [ 6329.794361] LNet: 340021:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6329.797033] LNetError: 340021:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6329.804338] LNet: Removed LNI 192.168.203.122@tcp [ 6330.120766] Key type .llcrypt unregistered [ 6330.121829] Key type ._llcrypt unregistered [ 6342.039541] Key type ._llcrypt registered [ 6342.040700] Key type .llcrypt registered [ 6342.086127] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 6348.664279] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 6349.244075] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6349.252091] alg: No test for adler32 (adler32-zlib) [ 6350.111254] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 6350.204073] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 6351.791219] Key type lgssc registered [ 6352.188438] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6355.363537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6357.575396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6359.728483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6361.992862] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6364.330388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6364.351029] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6364.355486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6365.440972] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6365.453802] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6365.500198] Lustre: lustre-MDT0000: new disk, initializing [ 6365.526857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6365.534635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6366.893904] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6371.079951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6371.099310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6371.132072] Lustre: 343942:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6371.156552] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6371.158393] Lustre: Skipped 1 previous similar message [ 6371.194733] Lustre: lustre-MDT0001: new disk, initializing [ 6371.228429] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6371.241538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6371.245907] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6372.510478] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6375.539832] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6376.793120] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6379.084101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6379.112325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6379.240249] Lustre: lustre-OST0000: new disk, initializing [ 6379.243496] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6379.278862] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6380.802424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6380.807217] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6380.856251] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6381.276569] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6384.427095] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6385.826603] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6385.900449] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6387.266823] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6387.339873] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6391.264765] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6391.265894] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6391.267059] LustreError: Skipped 1 previous similar message [ 6391.271416] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6391.275831] Lustre: Skipped 1 previous similar message [ 6394.030180] LustreError: 346881:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6394.074715] Lustre: server umount lustre-OST0000 complete [ 6396.384952] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6396.385754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6396.388570] Lustre: Skipped 2 previous similar messages [ 6396.394091] Lustre: Skipped 1 previous similar message [ 6401.470660] LustreError: 347082:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6401.473225] LustreError: 347082:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6401.505052] Lustre: server umount lustre-MDT0000 complete [ 6402.772293] LustreError: 344724:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763159653 with bad export cookie 13839837263277246580 [ 6402.775611] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6402.775769] LustreError: 344724:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6402.840328] LustreError: 347283:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6402.843363] LustreError: 347283:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6402.898055] Lustre: server umount lustre-MDT0001 complete [ 6405.871383] Lustre: DEBUG MARKER: == conf-sanity test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 17:34:15 (1763159655) [ 6407.802538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6409.734839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6417.186561] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 6420.284647] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 6423.714278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6425.987556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6427.978896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6430.097740] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6432.396640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6432.422296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6432.499300] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6432.510061] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6432.551024] Lustre: lustre-MDT0000: new disk, initializing [ 6432.574874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6432.581633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6433.889254] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6438.172625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6438.197379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6438.236597] Lustre: 351049:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6438.265270] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6438.266940] Lustre: Skipped 1 previous similar message [ 6438.308224] Lustre: lustre-MDT0001: new disk, initializing [ 6438.345306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6438.358506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6438.363272] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6439.709395] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6442.782718] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6443.994315] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6446.259167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6446.286643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6446.375602] Lustre: lustre-OST0000: new disk, initializing [ 6446.377461] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6446.402204] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6448.262985] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6449.210489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6449.214386] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6449.243377] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6451.481434] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6452.918669] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6453.001511] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6454.377281] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6454.480567] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6459.362474] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6459.362927] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6459.367581] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6459.372043] Lustre: Skipped 1 previous similar message [ 6461.230099] LustreError: 353981:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6461.232169] LustreError: 353981:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6461.272173] Lustre: server umount lustre-OST0000 complete [ 6463.969442] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6463.971712] LustreError: Skipped 1 previous similar message [ 6463.973061] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6463.976336] Lustre: Skipped 1 previous similar message [ 6463.978214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6468.678903] LustreError: 354183:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6468.680664] LustreError: 354183:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6468.713234] Lustre: server umount lustre-MDT0000 complete [ 6469.601203] LustreError: 351059:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6469.606565] LustreError: 351059:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6469.995210] LustreError: 354184:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763159720 with bad export cookie 13839837263277247448 [ 6469.997307] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6470.000416] LustreError: 354184:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6470.130236] Lustre: server umount lustre-MDT0001 complete [ 6472.996729] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 17:35:22 (1763159722) [ 6475.476837] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 17:35:25 (1763159725) [ 6478.282183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6478.454185] LustreError: 355399:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6478.482459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6479.830631] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6482.753547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6484.238510] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6486.206305] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6487.426653] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6489.756800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6491.864585] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6493.613604] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6507.522454] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 6508.096250] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 6508.686152] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 6509.239989] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 6509.902772] Lustre: Failing over lustre-MDT0000 [ 6510.055808] LustreError: 357857:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6510.057639] LustreError: 357857:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6510.084204] Lustre: server umount lustre-MDT0000 complete [ 6513.634292] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6513.635639] LustreError: 355398:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6513.639125] Lustre: Skipped 3 previous similar messages [ 6513.639405] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6513.647731] LustreError: 355398:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6518.753123] LustreError: 355399:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6518.759190] LustreError: 355399:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6523.073252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6523.111567] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6523.214063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6523.217489] Lustre: Skipped 2 previous similar messages [ 6523.234886] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6524.536864] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6524.623782] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6528.486677] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6528.488382] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 6528.506718] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6529.637449] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6530.176945] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6531.991295] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 6532.655396] Lustre: Failing over lustre-MDT0000 [ 6532.832143] LustreError: 358937:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6532.833996] LustreError: 358937:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6532.862049] Lustre: server umount lustre-MDT0000 complete [ 6533.601623] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6533.602760] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6533.603018] LustreError: 355395:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6533.603024] LustreError: 355395:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6533.604790] Lustre: Skipped 2 previous similar messages [ 6543.840231] LustreError: 355399:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6543.849874] LustreError: 355399:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 6545.787060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6545.826845] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6545.943653] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6547.289406] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6550.222806] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6551.015831] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6551.018098] Lustre: Skipped 2 previous similar messages [ 6551.023967] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 6551.041688] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6552.226786] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6552.766911] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6556.129780] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6556.131330] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6556.133022] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6556.133028] Lustre: Skipped 2 previous similar messages [ 6556.139025] Lustre: Skipped 1 previous similar message [ 6561.183402] Lustre: server umount lustre-MDT0000 complete [ 6561.249609] LustreError: 355399:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6561.255845] LustreError: 355399:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6562.436534] LustreError: 355380:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763159812 with bad export cookie 13839837263277250087 [ 6562.441762] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6562.570228] Lustre: server umount lustre-MDT0001 complete [ 6574.130205] LustreError: 360368:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6574.132124] LustreError: 360368:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 6574.173320] Lustre: server umount lustre-OST0000 complete [ 6579.295080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6580.786893] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6583.647916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6585.139864] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6587.018588] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6588.269166] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6590.635786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6590.748574] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6590.750460] Lustre: Skipped 3 previous similar messages [ 6592.668340] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6594.458328] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6604.070879] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6609.377302] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6609.379963] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6609.381521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6609.383786] Lustre: Skipped 2 previous similar messages [ 6609.387586] Lustre: Skipped 3 previous similar messages [ 6612.303064] Lustre: server umount lustre-OST0000 complete [ 6619.674836] Lustre: server umount lustre-MDT0000 complete [ 6620.914203] LustreError: 361176:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763159871 with bad export cookie 13839837263277250815 [ 6620.917100] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6620.917611] LustreError: 361176:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 6621.035643] Lustre: server umount lustre-MDT0001 complete [ 6624.544388] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 6625.562991] Key type lgssc unregistered [ 6625.685459] LNet: 364394:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6625.688081] LNetError: 364394:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6625.696323] LNet: Removed LNI 192.168.203.122@tcp [ 6626.001070] Key type .llcrypt unregistered [ 6626.002167] Key type ._llcrypt unregistered [ 6635.735579] Key type ._llcrypt registered [ 6635.736551] Key type .llcrypt registered [ 6635.777367] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 6636.134820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6636.204830] alg: No test for adler32 (adler32-zlib) [ 6637.074131] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 6637.161877] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 6638.743119] Key type lgssc registered [ 6639.143967] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6641.805224] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6641.811228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6642.971859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6644.208472] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6646.974783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6647.096974] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6648.355333] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6650.061824] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6651.291970] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6653.491884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6653.612828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6655.608256] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6657.433470] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6667.940704] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6672.863961] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6672.867138] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6672.871649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6673.377944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6673.379443] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6677.885281] LustreError: 368136:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6677.913223] Lustre: server umount lustre-MDT0000 complete [ 6678.496679] LustreError: 367280:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6678.502282] LustreError: 367280:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6679.194347] LustreError: 367489:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763159929 with bad export cookie 17175937223356501787 [ 6679.200111] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6679.200109] LustreError: 367489:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6679.267502] LustreError: 368339:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6679.269229] LustreError: 368339:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6679.315491] Lustre: server umount lustre-MDT0001 complete [ 6690.802113] LustreError: 368539:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6690.804723] LustreError: 368539:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6690.843908] Lustre: server umount lustre-OST0000 complete [ 6695.194627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6695.342864] LustreError: 369269:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6695.366599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6696.594701] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6699.530648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6699.646659] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6700.915360] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6702.637884] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6703.839121] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6706.140564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6708.148096] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6709.930346] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6719.590069] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 6730.210063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6730.211920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6730.214581] Lustre: Skipped 3 previous similar messages [ 6730.217359] Lustre: Skipped 4 previous similar messages [ 6733.307400] LustreError: 371634:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6733.309255] LustreError: 371634:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6733.331665] Lustre: server umount lustre-MDT0000 complete [ 6734.539578] LustreError: 370884:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763159985 with bad export cookie 17175937223356502886 [ 6734.542568] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6734.545327] LustreError: 370884:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6734.659134] Lustre: server umount lustre-MDT0001 complete [ 6746.163061] LustreError: 372038:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6746.165048] LustreError: 372038:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 6746.205445] Lustre: server umount lustre-OST0000 complete [ 6751.370966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6751.545869] LustreError: 372866:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6751.571935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6751.574941] Lustre: Skipped 1 previous similar message [ 6752.871216] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6755.798895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6757.184019] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6759.190471] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6760.400873] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6762.642373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6764.626566] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6766.445673] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6777.127151] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 6786.528926] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6786.531038] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6786.534967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6787.554446] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6789.358316] LustreError: 375133:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6789.360199] LustreError: 375133:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6789.399118] Lustre: server umount lustre-OST0000 complete [ 6791.647784] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6791.650026] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6791.653505] Lustre: Skipped 1 previous similar message [ 6791.655233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6791.657678] Lustre: Skipped 2 previous similar messages [ 6796.754623] Lustre: server umount lustre-MDT0000 complete [ 6797.792127] LustreError: 373589:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6797.797493] LustreError: 373589:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6797.988510] LustreError: 372890:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763160048 with bad export cookie 17175937223356503894 [ 6797.991404] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6797.992030] LustreError: 372890:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6798.123418] Lustre: server umount lustre-MDT0001 complete [ 6801.521680] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 6802.532712] Key type lgssc unregistered [ 6802.655376] LNet: 376069:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6802.658147] LNetError: 376069:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6802.669428] LNet: Removed LNI 192.168.203.122@tcp [ 6802.992992] Key type .llcrypt unregistered [ 6802.994115] Key type ._llcrypt unregistered [ 6812.520429] Key type ._llcrypt registered [ 6812.521757] Key type .llcrypt registered [ 6812.567659] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 6812.901769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6812.951989] alg: No test for adler32 (adler32-zlib) [ 6813.831351] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 6813.931286] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 6815.519138] Key type lgssc registered [ 6815.847225] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6818.504416] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6818.512032] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6818.514485] LustreError: 377465:0:(osd_handler.c:8577:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 6818.523119] LustreError: 377465:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 6818.525247] LustreError: 377465:0:(obd_mount.c:193:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 6818.527340] LustreError: 377465:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 6818.529726] LustreError: 377465:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6820.917794] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6820.920518] LustreError: 377862:0:(osd_handler.c:8577:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 6820.931148] LustreError: 377862:0:(obd_config.c:783:class_setup()) setup lustre-OST0000-osd failed (-22) [ 6820.933466] LustreError: 377862:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 6820.936358] LustreError: 377862:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 6820.939532] LustreError: 377862:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6827.087628] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 6828.097709] Key type lgssc unregistered [ 6828.215391] LNet: 378847:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6828.218215] LNetError: 378847:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6828.227308] LNet: Removed LNI 192.168.203.122@tcp [ 6828.546506] Key type .llcrypt unregistered [ 6828.547557] Key type ._llcrypt unregistered [ 6834.569698] Key type ._llcrypt registered [ 6834.571472] Key type .llcrypt registered [ 6834.612742] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 6840.668213] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 6841.057903] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6841.105081] alg: No test for adler32 (adler32-zlib) [ 6841.967106] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 6842.051830] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 6843.631189] Key type lgssc registered [ 6844.007102] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6846.951298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6849.062983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6851.061590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6853.326698] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6855.620462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6855.638525] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6855.642787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6856.730456] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6856.740863] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6856.776803] Lustre: lustre-MDT0000: new disk, initializing [ 6856.799899] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6856.806908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6858.033803] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6862.078714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6862.102905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6862.136418] Lustre: 381949:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6862.161277] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6862.163682] Lustre: Skipped 1 previous similar message [ 6862.201338] Lustre: lustre-MDT0001: new disk, initializing [ 6862.234499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6862.246938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6862.249870] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6863.579640] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6866.723853] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6867.939267] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6870.296022] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6870.313947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6870.427693] Lustre: lustre-OST0000: new disk, initializing [ 6870.429901] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6870.456788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6872.337201] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6875.362366] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6876.726377] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6877.186516] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6877.189874] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6877.237390] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6877.891230] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 6879.313207] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6879.388312] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6882.273244] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6882.275581] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6882.276715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6882.280348] Lustre: Skipped 1 previous similar message [ 6886.125103] LustreError: 384897:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6886.166637] Lustre: server umount lustre-OST0000 complete [ 6887.904839] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6887.909873] LustreError: Skipped 1 previous similar message [ 6887.912546] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6887.920817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6887.924254] Lustre: Skipped 1 previous similar message [ 6892.513071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6892.514110] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6893.572339] LustreError: 385098:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6893.574729] LustreError: 385098:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6893.603272] Lustre: server umount lustre-MDT0000 complete [ 6895.079251] LustreError: 382730:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763160145 with bad export cookie 2247446567009631973 [ 6895.082297] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6895.082437] LustreError: 382730:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6895.152650] LustreError: 385299:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6895.154818] LustreError: 385299:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6895.206216] Lustre: server umount lustre-MDT0001 complete [ 6898.065528] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 6898.643471] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 17:42:28 (1763160148) [ 6900.812456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6900.991168] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6902.240637] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6905.349964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6905.473237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6906.708265] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6908.641516] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6910.193350] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6912.426763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6914.450514] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6916.241328] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6918.647667] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6918.676575] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6918.747298] Lustre: lustre-OST0001: new disk, initializing [ 6918.749183] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6918.786127] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6918.788022] Lustre: Skipped 1 previous similar message [ 6919.934932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6919.938136] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6919.969412] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6920.767217] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6923.851494] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6926.755315] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6926.758123] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6926.759429] Lustre: Skipped 1 previous similar message [ 6926.763530] Lustre: Skipped 1 previous similar message [ 6931.825306] LustreError: 389215:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 6931.827419] LustreError: 389215:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6931.849411] Lustre: server umount lustre-OST0001 complete [ 6931.940977] LustreError: 389216:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6931.944964] LustreError: 389216:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6937.057994] LustreError: 387613:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6937.064571] LustreError: 387613:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6942.180078] LustreError: 387614:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6942.185060] LustreError: 387614:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6945.247754] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6945.249781] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6945.253185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6945.255537] Lustre: Skipped 2 previous similar messages [ 6947.298483] LustreError: 387613:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6947.299923] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6947.302572] LustreError: 387613:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6949.998098] LustreError: 389438:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6949.999915] LustreError: 389438:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6950.033874] Lustre: server umount lustre-OST0000 complete [ 6951.903827] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6957.400677] Lustre: server umount lustre-MDT0000 complete [ 6957.536292] LustreError: 386016:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6957.544111] LustreError: 386016:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6958.681706] LustreError: 385996:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763160209 with bad export cookie 2247446567009632841 [ 6958.684527] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6958.685528] LustreError: 385996:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6964.801311] LustreError: 389842:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 6964.804216] LustreError: 389842:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 6964.865833] Lustre: server umount lustre-MDT0001 complete [ 6968.695285] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 6969.821582] Key type lgssc unregistered [ 6969.954372] LNet: 390373:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6969.956662] LNetError: 390373:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6969.965411] LNet: Removed LNI 192.168.203.122@tcp [ 6970.309668] Key type .llcrypt unregistered [ 6970.310805] Key type ._llcrypt unregistered [ 6981.100262] Key type ._llcrypt registered [ 6981.101758] Key type .llcrypt registered [ 6981.138274] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 6987.317343] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 6987.598777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6987.630079] alg: No test for adler32 (adler32-zlib) [ 6988.494513] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 6988.583257] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 6990.167185] Key type lgssc registered [ 6990.518817] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6993.338762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6995.448944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6997.431280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6999.470378] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7001.915106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7001.929702] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7001.935263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7003.038480] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7003.053354] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7003.092887] Lustre: lustre-MDT0000: new disk, initializing [ 7003.118406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7003.126207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7004.510541] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7008.686246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7008.710384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7008.743306] Lustre: 394294:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7008.765675] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7008.767222] Lustre: Skipped 1 previous similar message [ 7008.801898] Lustre: lustre-MDT0001: new disk, initializing [ 7008.831498] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7008.842287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7008.845089] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7010.096446] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7013.304313] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7014.529612] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7016.943066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7016.968272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7017.070772] Lustre: lustre-OST0000: new disk, initializing [ 7017.073101] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7017.101831] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7018.985576] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7021.989397] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7022.587197] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7022.590106] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7022.625978] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7023.429829] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7023.505705] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7024.803613] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7024.876189] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7027.680848] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7027.681467] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7027.683584] LustreError: Skipped 1 previous similar message [ 7027.684258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7027.690062] Lustre: Skipped 1 previous similar message [ 7031.533981] LustreError: 397232:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7031.564868] Lustre: server umount lustre-OST0000 complete [ 7032.800471] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7032.800992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7032.807398] Lustre: Skipped 1 previous similar message [ 7032.809225] Lustre: Skipped 1 previous similar message [ 7037.920244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7037.921945] Lustre: Skipped 2 previous similar messages [ 7043.040107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7043.041835] Lustre: Skipped 1 previous similar message [ 7047.135098] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7047.166259] LustreError: 397433:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7047.168105] LustreError: 397433:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7047.191991] Lustre: server umount lustre-MDT0000 complete [ 7048.374050] LustreError: 394286:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763160298 with bad export cookie 520820966116537692 [ 7048.376338] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7048.378764] LustreError: 394286:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7048.440672] LustreError: 397636:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7048.442676] LustreError: 397636:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7048.486544] Lustre: server umount lustre-MDT0001 complete [ 7051.182238] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 17:45:00 (1763160300) [ 7054.254093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7057.053605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7057.213081] Lustre: lustre-MDT0000: new disk, initializing [ 7057.225777] LustreError: 398901:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7057.232357] LustreError: 398901:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7057.251810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7058.666170] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7061.541796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7061.654152] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7062.933397] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7064.644764] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7065.758259] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7066.424209] LustreError: 398931:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7066.427743] LustreError: 398931:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7066.431522] LustreError: 398931:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 7067.104914] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7067.105073] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7067.105785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7067.110720] Lustre: Skipped 1 previous similar message [ 7072.573750] LustreError: 400245:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7072.575536] LustreError: 400245:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7072.598477] Lustre: server umount lustre-MDT0000 complete [ 7073.784119] LustreError: 398882:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763160324 with bad export cookie 520820966116538574 [ 7073.787325] LustreError: 398882:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7073.787674] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7073.897777] Lustre: server umount lustre-MDT0001 complete [ 7077.001411] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 17:45:26 (1763160326) [ 7079.194920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7079.347075] LustreError: 401166:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7079.369717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7080.617972] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7083.373924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7084.774704] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7086.756957] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7087.928707] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7090.205208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7092.029760] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7093.933405] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7106.897680] Lustre: Permanently deactivating lustre-OST0000 [ 7106.903414] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7113.204656] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7114.313313] LustreError: 403878:0:(mgs_llog.c:1938:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 7114.320650] LustreError: 403878:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [ 7114.325410] LustreError: 403878:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7115.023327] LustreError: 403926:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [ 7115.025852] LustreError: 403926:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7121.771205] LustreError: 404023:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7121.774183] LustreError: 404023:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 7121.806967] Lustre: server umount lustre-OST0000 complete [ 7122.885311] LustreError: 404175:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [ 7122.889641] LustreError: 404175:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7124.451547] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7124.451877] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7124.452649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7124.452658] Lustre: Skipped 3 previous similar messages [ 7124.459599] Lustre: Skipped 1 previous similar message [ 7138.271163] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7138.310858] LustreError: 404273:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7138.314645] LustreError: 404273:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7138.339590] Lustre: server umount lustre-MDT0000 complete [ 7139.809746] LustreError: 401165:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7139.818526] LustreError: 401165:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7139.920400] LustreError: 401145:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763160390 with bad export cookie 520820966116539106 [ 7139.923659] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7139.927312] LustreError: 401145:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7149.584992] Lustre: server umount lustre-MDT0001 complete [ 7152.450209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7154.117766] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7155.168456] LustreError: 405316:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 7157.761929] Lustre: 405654:0:(mgs_llog.c:1699:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.122@tcp [ 7158.473493] LustreError: 405751:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7158.475341] LustreError: 405751:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 7158.501358] Lustre: server umount MGS complete [ 7162.359077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7162.535321] LustreError: 406379:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7162.542184] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7162.544342] Lustre: Skipped 1 previous similar message [ 7162.566789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7162.568527] Lustre: Skipped 2 previous similar messages [ 7163.990437] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7167.065970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7167.163288] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7168.510273] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7170.334949] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7171.535411] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7172.210686] Lustre: Permanently reactivating lustre-OST0000 [ 7172.215850] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7172.218609] Lustre: Skipped 2 previous similar messages [ 7198.175187] LustreError: 407967:0:(osp_dev.c:692:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 7200.905438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7201.033755] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7201.035990] Lustre: Skipped 1 previous similar message [ 7202.924434] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7204.752575] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7218.664917] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7218.665108] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7218.668908] LustreError: Skipped 1 previous similar message [ 7218.670320] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7218.676802] Lustre: Skipped 1 previous similar message [ 7218.679242] Lustre: Skipped 5 previous similar messages [ 7222.635189] LustreError: 409762:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7222.636944] LustreError: 409762:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 7222.667909] Lustre: server umount lustre-OST0000 complete [ 7228.897281] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7228.897435] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7228.904574] Lustre: Skipped 1 previous similar message [ 7230.041882] Lustre: server umount lustre-MDT0000 complete [ 7231.353368] LustreError: 407730:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763160481 with bad export cookie 520820966116540051 [ 7231.355332] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7231.356650] LustreError: 407730:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7231.481706] Lustre: server umount lustre-MDT0001 complete [ 7235.113307] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 7236.181807] Key type lgssc unregistered [ 7236.322455] LNet: 410701:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7236.324904] LNetError: 410701:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7236.331443] LNet: Removed LNI 192.168.203.122@tcp [ 7236.674542] Key type .llcrypt unregistered [ 7236.675867] Key type ._llcrypt unregistered [ 7242.717974] Key type ._llcrypt registered [ 7242.719307] Key type .llcrypt registered [ 7242.764047] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 7248.987798] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 7249.390224] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7249.396389] alg: No test for adler32 (adler32-zlib) [ 7250.239096] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 7250.318248] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 7251.895179] Key type lgssc registered [ 7252.228870] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7255.325332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7257.914358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7260.016166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7262.116977] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7264.519130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7264.538048] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7264.542488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7265.632866] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7265.644926] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7265.683537] Lustre: lustre-MDT0000: new disk, initializing [ 7265.711132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7265.720452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7267.010788] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7271.074590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7271.096724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7271.130374] Lustre: 413806:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7271.152375] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7271.154222] Lustre: Skipped 1 previous similar message [ 7271.192194] Lustre: lustre-MDT0001: new disk, initializing [ 7271.225184] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7271.237885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7271.243129] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7272.541437] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7275.666153] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7276.878580] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7279.232444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7279.257057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7279.352824] Lustre: lustre-OST0000: new disk, initializing [ 7279.354656] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7279.386946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7281.299701] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7283.394801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7283.400860] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7283.437026] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7284.455201] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7285.779660] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7285.849453] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7287.115279] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7287.201853] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7288.802112] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7288.802946] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7288.804820] LustreError: Skipped 1 previous similar message [ 7288.808494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7288.811487] Lustre: Skipped 1 previous similar message [ 7289.902069] LustreError: 416744:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7289.938023] Lustre: server umount lustre-OST0000 complete [ 7291.871911] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7291.874064] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7291.877852] Lustre: Skipped 1 previous similar message [ 7291.879662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7293.920729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7293.921227] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7297.281724] LustreError: 416946:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7297.283898] LustreError: 416946:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7297.314713] Lustre: server umount lustre-MDT0000 complete [ 7298.519130] LustreError: 413799:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763160549 with bad export cookie 11396399290267381839 [ 7298.522715] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7298.525395] LustreError: 413799:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7298.597051] LustreError: 417148:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7298.600394] LustreError: 417148:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7298.654254] Lustre: server umount lustre-MDT0001 complete [ 7301.655290] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 7302.228491] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 17:49:11 (1763160551) [ 7304.942455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7305.123584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7306.506417] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7309.435049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7309.570467] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7310.977649] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7312.898415] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7314.098842] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7316.381204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7318.378293] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7320.173356] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7320.843196] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 7331.601232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 7334.881531] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7334.881880] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7334.884419] LustreError: Skipped 1 previous similar message [ 7334.888737] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7334.891600] Lustre: Skipped 2 previous similar messages [ 7339.117731] LustreError: 420325:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7339.119636] LustreError: 420325:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7339.154839] Lustre: server umount lustre-OST0000 complete [ 7340.511899] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7340.515918] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7340.522099] Lustre: Skipped 1 previous similar message [ 7340.524746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7346.429811] LustreError: 420528:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7346.432617] LustreError: 420528:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7346.460787] Lustre: server umount lustre-MDT0000 complete [ 7347.683098] LustreError: 417846:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763160598 with bad export cookie 11396399290267382707 [ 7347.687254] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7347.688403] LustreError: 417846:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7347.812709] Lustre: server umount lustre-MDT0001 complete [ 7351.239320] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 7352.311301] Key type lgssc unregistered [ 7352.438434] LNet: 421260:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7352.442103] LNetError: 421260:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7352.450355] LNet: Removed LNI 192.168.203.122@tcp [ 7352.752218] Key type .llcrypt unregistered [ 7352.753302] Key type ._llcrypt unregistered [ 7360.584499] Key type ._llcrypt registered [ 7360.585818] Key type .llcrypt registered [ 7360.625466] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 7360.722436] Key type .llcrypt unregistered [ 7360.723494] Key type ._llcrypt unregistered [ 7367.508511] Key type ._llcrypt registered [ 7367.509467] Key type .llcrypt registered [ 7367.543751] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 7367.905843] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7367.937630] alg: No test for adler32 (adler32-zlib) [ 7368.793041] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 7368.871692] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 7370.447150] Key type lgssc registered [ 7370.821623] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7373.455348] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7373.461285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7374.606761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7375.822969] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7379.055387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7379.241277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7381.214764] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7382.989263] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7384.545696] LustreError: 423406:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7384.550080] LustreError: 423406:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7385.169783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7385.317744] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7386.341857] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7386.744448] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7390.689338] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7390.695091] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7390.704541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7391.713680] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7391.717581] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7391.718593] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7391.726908] Lustre: Skipped 1 previous similar message [ 7395.698122] LustreError: 425465:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 7395.747756] Lustre: server umount lustre-OST0000 complete [ 7399.905384] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 7399.908185] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7399.912921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7401.952390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7401.952844] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7403.347657] LustreError: 425667:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7403.351609] LustreError: 425667:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7403.385649] Lustre: server umount lustre-MDT0000 complete [ 7404.802967] LustreError: 423388:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763160655 with bad export cookie 4643893781612362785 [ 7404.809540] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7404.809547] LustreError: 423388:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7404.903279] LustreError: 425868:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7404.908761] LustreError: 425868:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7404.964374] Lustre: server umount lustre-MDT0001 complete [ 7408.881360] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 7410.001993] Key type lgssc unregistered [ 7410.133607] LNet: 426398:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7410.136290] LNetError: 426398:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7410.144412] LNet: Removed LNI 192.168.203.122@tcp [ 7410.523225] Key type .llcrypt unregistered [ 7410.524543] Key type ._llcrypt unregistered [ 7420.388400] Key type ._llcrypt registered [ 7420.389475] Key type .llcrypt registered [ 7420.429639] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 7420.749442] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7420.790265] alg: No test for adler32 (adler32-zlib) [ 7421.653730] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 7421.749343] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 7423.343140] Key type lgssc registered [ 7423.805126] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7426.488840] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7426.495229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7441.951361] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7447.071179] LustreError: 427746:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.122@tcp: failed processing log, type 4: rc = -110 [ 7473.695126] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7475.496867] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7477.265110] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7479.501689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7479.674098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7479.691813] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 7480.892550] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7483.657627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7483.791904] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7485.086759] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7486.864817] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7488.140350] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7493.090228] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7493.092099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7493.093679] Lustre: Skipped 1 previous similar message [ 7496.622328] LustreError: 430026:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 7496.658376] Lustre: server umount lustre-OST0000 complete [ 7499.231873] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7499.234243] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7499.237989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7499.239539] Lustre: Skipped 1 previous similar message [ 7503.328417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7503.328858] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7504.378336] LustreError: 430229:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 7504.381339] LustreError: 430229:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7504.407803] Lustre: server umount lustre-MDT0000 complete [ 7505.564149] LustreError: 427754:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763160756 with bad export cookie 9726627160618591451 [ 7505.566760] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7505.567394] LustreError: 427754:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7505.637197] LustreError: 430433:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7505.639136] LustreError: 430433:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7505.688065] Lustre: server umount lustre-MDT0001 complete [ 7509.514434] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 7510.573841] Key type lgssc unregistered [ 7510.710504] LNet: 430964:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7510.713198] LNetError: 430964:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7510.721482] LNet: Removed LNI 192.168.203.122@tcp [ 7511.009507] Key type .llcrypt unregistered [ 7511.010647] Key type ._llcrypt unregistered [ 7520.647453] Key type ._llcrypt registered [ 7520.649290] Key type .llcrypt registered [ 7520.690171] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 7521.241245] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7521.250816] alg: No test for adler32 (adler32-zlib) [ 7522.100301] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 7522.184088] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 7523.759163] Key type lgssc registered [ 7524.113589] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7526.635553] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7526.641288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7527.794923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7529.117315] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7532.118256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7532.245770] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7533.783638] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7536.083801] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7537.433131] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7540.479722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7540.651526] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7542.769617] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7545.060462] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7553.968876] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 7558.111958] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7558.114029] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7558.117803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7559.137379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7559.138064] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7559.139080] Lustre: Skipped 1 previous similar message [ 7559.143613] Lustre: Skipped 1 previous similar message [ 7560.917769] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.22@tcp (stopping) [ 7560.919875] Lustre: Skipped 1 previous similar message [ 7562.885481] LustreError: 434605:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7562.914616] Lustre: server umount lustre-MDT0000 complete [ 7563.870063] LustreError: 432319:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.22@tcp arrived at 1763160814 with bad export cookie 4291393509230418465 [ 7564.256902] LustreError: 432334:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7564.265396] LustreError: 432334:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7564.511174] LustreError: 432319:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763160815 with bad export cookie 4291393509230417814 [ 7564.515057] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7564.515109] LustreError: 432319:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7567.327982] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7567.333152] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7567.341971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7569.375842] LustreError: 433060:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7569.380766] LustreError: 433060:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7570.603086] LustreError: 434808:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7570.605458] LustreError: 434808:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7570.637416] Lustre: server umount lustre-OST0000 complete [ 7573.227441] LustreError: 435162:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7573.229798] LustreError: 435162:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7573.307262] Lustre: server umount lustre-MDT0001 complete [ 7577.024286] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 7578.206507] Key type lgssc unregistered [ 7578.367854] LNet: 435693:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7578.372749] LNetError: 435693:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7578.382605] LNet: Removed LNI 192.168.203.122@tcp [ 7578.776928] Key type .llcrypt unregistered [ 7578.779290] Key type ._llcrypt unregistered [ 7588.162639] Key type ._llcrypt registered [ 7588.163616] Key type .llcrypt registered [ 7588.198233] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 7588.590247] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7588.600395] alg: No test for adler32 (adler32-zlib) [ 7589.448497] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 7589.532377] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 7591.119286] Key type lgssc registered [ 7591.710035] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7594.550154] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7594.558743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7595.790790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7597.385538] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7600.960885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7601.168868] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7602.820548] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7605.365700] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7606.652343] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7608.841852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7609.000120] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7611.064581] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7612.773124] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7622.314066] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 7626.719903] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 7626.721970] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7626.725568] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7627.746596] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7627.746850] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7630.397183] LustreError: 439331:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7630.421115] Lustre: server umount lustre-MDT0001 complete [ 7632.864662] LustreError: 437791:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7632.865793] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7632.865820] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7632.866771] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7632.866775] Lustre: Skipped 2 previous similar messages [ 7632.868707] LustreError: 437791:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7637.986891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7637.987117] LustreError: 437791:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7638.133182] LustreError: 439535:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7638.137214] LustreError: 439535:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7638.189627] Lustre: server umount lustre-OST0000 complete [ 7639.826781] LustreError: 439738:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7639.830453] LustreError: 439738:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7639.913334] Lustre: server umount lustre-MDT0000 complete [ 7645.144460] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 7646.675894] Key type lgssc unregistered [ 7646.857931] LNet: 440422:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7646.862741] LNetError: 440422:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7646.876746] LNet: Removed LNI 192.168.203.122@tcp [ 7647.347143] Key type .llcrypt unregistered [ 7647.348988] Key type ._llcrypt unregistered [ 7657.790630] Key type ._llcrypt registered [ 7657.791914] Key type .llcrypt registered [ 7657.835990] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 7663.719830] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 7664.024500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7664.041083] alg: No test for adler32 (adler32-zlib) [ 7664.902992] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 7665.001851] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 7666.591172] Key type lgssc registered [ 7667.147680] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7669.932983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7672.507749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7674.464480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7677.313029] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7681.400354] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 7685.524524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7688.694257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7688.720992] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7688.728246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7689.858994] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7689.875568] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7689.938463] Lustre: lustre-MDT0000: new disk, initializing [ 7689.981387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7689.995677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7691.751892] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7696.069118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7698.509759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7698.540751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7698.637281] Lustre: lustre-OST0000: new disk, initializing [ 7698.639746] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7698.641582] Lustre: Skipped 1 previous similar message [ 7698.669620] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7700.820453] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7704.007371] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7704.068369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 7704.070703] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 7704.078343] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 7707.708140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7710.043349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7710.078299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7710.136816] Lustre: 444370:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7710.187856] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7710.238139] Lustre: lustre-MDT0001: new disk, initializing [ 7710.298804] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7710.318791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 7710.324300] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 7711.842966] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7715.669483] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7715.744396] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7717.272376] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7717.379368] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7721.685839] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.22@tcp (stopping) [ 7722.468948] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 7722.471243] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7722.473394] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7722.477639] Lustre: Skipped 1 previous similar message [ 7725.763036] LustreError: 447790:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7725.785094] Lustre: server umount lustre-MDT0001 complete [ 7732.706758] LustreError: 447793:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7732.711029] LustreError: 447793:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 7737.828393] LustreError: 444375:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7737.832243] LustreError: 444375:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7739.871927] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7739.873927] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7739.877348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7739.878845] Lustre: Skipped 1 previous similar message [ 7742.944161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7742.944895] LustreError: 444379:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7744.237088] LustreError: 448063:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 7744.239583] LustreError: 448063:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7744.271726] Lustre: server umount lustre-OST0000 complete [ 7751.695190] LustreError: 448264:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 7751.698475] LustreError: 448264:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7751.781089] Lustre: server umount lustre-MDT0000 complete [ 7756.319393] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 7757.298473] Key type lgssc unregistered [ 7757.418388] LNet: 448948:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7757.420612] LNetError: 448948:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7757.427327] LNet: Removed LNI 192.168.203.122@tcp [ 7757.694557] Key type .llcrypt unregistered [ 7757.695765] Key type ._llcrypt unregistered [ 7769.855998] Key type ._llcrypt registered [ 7769.857747] Key type .llcrypt registered [ 7769.898962] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 7770.001464] Key type .llcrypt unregistered [ 7770.003272] Key type ._llcrypt unregistered [ 7776.105328] Key type ._llcrypt registered [ 7776.106502] Key type .llcrypt registered [ 7776.144235] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 7776.429925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7776.459040] alg: No test for adler32 (adler32-zlib) [ 7777.310194] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 7777.388191] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 7778.959203] Key type lgssc registered [ 7779.370694] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7781.885755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7783.792246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7786.102351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7788.370644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7788.389954] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7788.395189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7789.473115] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7789.485284] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7789.527638] Lustre: lustre-MDT0000: new disk, initializing [ 7789.551970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7789.559148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7790.802332] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7795.047615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7795.087111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7795.138755] Lustre: 452264:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7795.178272] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7795.181224] Lustre: Skipped 1 previous similar message [ 7795.226972] Lustre: lustre-MDT0001: new disk, initializing [ 7795.261524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7795.273173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7795.277975] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7796.498949] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7799.527929] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7800.595088] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7803.022490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7803.045954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7803.133703] Lustre: lustre-OST0000: new disk, initializing [ 7803.135672] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7803.161269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7804.903598] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7808.065460] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7810.549971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7810.552954] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7810.582780] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7815.650122] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7815.650527] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7815.651609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7815.653683] Lustre: Skipped 2 previous similar messages [ 7820.769585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7820.771553] Lustre: Skipped 4 previous similar messages [ 7821.779593] LustreError: 454705:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7821.818214] Lustre: server umount lustre-MDT0000 complete [ 7823.571165] LustreError: 452254:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763161074 with bad export cookie 2804189303557921861 [ 7823.577631] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7823.578238] LustreError: 452254:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7823.680626] LustreError: 454907:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7823.683922] LustreError: 454907:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7823.751352] Lustre: server umount lustre-MDT0001 complete [ 7835.384362] LustreError: 455107:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7835.388305] LustreError: 455107:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7835.459978] Lustre: server umount lustre-OST0000 complete [ 7840.095620] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 17:58:09 (1763161089) [ 7840.482071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7843.107288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7843.269775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7844.464379] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7847.691793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7847.825256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7849.137855] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7851.238302] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7852.850774] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7855.072549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7855.101056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7855.176429] Lustre: Found index 0 for lustre-OST0000, updating log [ 7856.937913] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7858.584165] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7867.558069] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 7872.992580] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7872.994136] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7872.995884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7872.998390] Lustre: Skipped 1 previous similar message [ 7876.331095] LustreError: 458216:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7876.332743] LustreError: 458216:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7876.363979] Lustre: server umount lustre-OST0000 complete [ 7877.238522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7878.623872] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7878.625776] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7878.629115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7878.630647] Lustre: Skipped 3 previous similar messages [ 7883.964250] LustreError: 458470:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7883.967848] LustreError: 458470:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7883.998915] Lustre: server umount lustre-MDT0000 complete [ 7885.240987] LustreError: 455914:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763161135 with bad export cookie 2804189303557923240 [ 7885.243950] LustreError: 455914:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7885.244861] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7885.358543] Lustre: server umount lustre-MDT0001 complete [ 7888.100741] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 17:58:57 (1763161137) [ 7888.416454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7888.707267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7889.021402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7889.328831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7889.637700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7889.951235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7890.285712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7890.610046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7890.905343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7891.196263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7891.489275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7891.795434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7892.107226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7892.394520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7892.693267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7892.984261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7893.281119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7893.572354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7893.872410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7894.169403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7894.467345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7894.771363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7895.077261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7895.364538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7895.650398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7895.943077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7896.239267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7896.527540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7896.825073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7897.118100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7897.399965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7897.682048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7897.975118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7898.278736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7898.566364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7898.866339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7899.159337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7899.450227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7899.764458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7900.064196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7900.352211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7900.659332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7900.968895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7901.274081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7901.584258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7901.901050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7902.218718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7902.537596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7902.845345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7903.157856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7903.455199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7905.997365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7906.144080] LustreError: 462054:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7906.149533] LustreError: 462054:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7906.166110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7906.168305] Lustre: Skipped 1 previous similar message [ 7907.259350] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7909.836380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7911.082737] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7912.782679] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7913.964679] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7916.147372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7916.172127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7916.256545] Lustre: Found index 0 for lustre-OST0000, updating log [ 7917.413409] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [ 7918.030806] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7920.405648] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7922.147391] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7922.149575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7922.154423] Lustre: Skipped 1 previous similar message [ 7922.162261] Lustre: Skipped 5 previous similar messages [ 7927.212719] LustreError: 464357:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7927.216262] LustreError: 464357:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 7927.251763] Lustre: server umount lustre-OST0000 complete [ 7929.505805] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7929.512208] Lustre: Skipped 1 previous similar message [ 7930.528080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7930.529653] Lustre: Skipped 5 previous similar messages [ 7934.610758] Lustre: server umount lustre-MDT0000 complete [ 7934.624113] LustreError: 462054:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7934.628018] LustreError: 462054:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7935.648184] LustreError: 462054:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7935.652218] LustreError: 462054:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7935.826379] LustreError: 463725:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763161186 with bad export cookie 2804189303557924304 [ 7935.829359] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7935.831473] LustreError: 463725:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7935.943231] Lustre: server umount lustre-MDT0001 complete [ 7936.826902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7942.570422] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 7946.387673] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 7950.613040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7952.598924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7955.093483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7957.206667] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7959.500494] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 18:00:09 (1763161209) [ 7993.176042] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 7994.161921] Key type lgssc unregistered [ 7994.282376] LNet: 468352:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7994.284425] LNetError: 468352:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8120.230410] LNet: Removed LNI 192.168.203.122@tcp [ 8120.516055] Key type .llcrypt unregistered [ 8120.517021] Key type ._llcrypt unregistered [ 8126.838149] Key type ._llcrypt registered [ 8126.839164] Key type .llcrypt registered [ 8126.871444] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 8127.157887] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8127.165482] alg: No test for adler32 (adler32-zlib) [ 8128.001610] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 8128.077051] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 8129.655114] Key type lgssc registered [ 8129.992400] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8131.393260] LNet: Added LNI 192.168.203.122@tcp100 [8/256/0/180] [ 8131.657577] LNet: Added LNI 192.168.203.122@tcp101 [8/256/0/180] [ 8131.935952] LNet: Added LNI 192.168.203.122@tcp102 [8/256/0/180] [ 8132.206891] LNet: Added LNI 192.168.203.122@tcp103 [8/256/0/180] [ 8132.478120] LNet: Added LNI 192.168.203.122@tcp104 [8/256/0/180] [ 8132.755956] LNet: Added LNI 192.168.203.122@tcp105 [8/256/0/180] [ 8133.026945] LNet: Added LNI 192.168.203.122@tcp106 [8/256/0/180] [ 8133.293486] LNet: Added LNI 192.168.203.122@tcp107 [8/256/0/180] [ 8133.574694] LNet: Added LNI 192.168.203.122@tcp108 [8/256/0/180] [ 8133.841990] LNet: Added LNI 192.168.203.122@tcp109 [8/256/0/180] [ 8134.106417] LNet: Added LNI 192.168.203.122@tcp110 [8/256/0/180] [ 8134.374453] LNet: Added LNI 192.168.203.122@tcp111 [8/256/0/180] [ 8134.639268] LNet: Added LNI 192.168.203.122@tcp112 [8/256/0/180] [ 8134.898676] LNet: Added LNI 192.168.203.122@tcp113 [8/256/0/180] [ 8135.165203] LNet: Added LNI 192.168.203.122@tcp114 [8/256/0/180] [ 8135.428983] LNet: Added LNI 192.168.203.122@tcp115 [8/256/0/180] [ 8135.696357] LNet: Added LNI 192.168.203.122@tcp116 [8/256/0/180] [ 8135.960456] LNet: Added LNI 192.168.203.122@tcp117 [8/256/0/180] [ 8136.227274] LNet: Added LNI 192.168.203.122@tcp118 [8/256/0/180] [ 8136.500627] LNet: Added LNI 192.168.203.122@tcp119 [8/256/0/180] [ 8136.796616] LNet: Added LNI 192.168.203.122@tcp120 [8/256/0/180] [ 8137.090297] LNet: Added LNI 192.168.203.122@tcp121 [8/256/0/180] [ 8137.380524] LNet: Added LNI 192.168.203.122@tcp122 [8/256/0/180] [ 8137.670849] LNet: Added LNI 192.168.203.122@tcp123 [8/256/0/180] [ 8137.942519] LNet: Added LNI 192.168.203.122@tcp124 [8/256/0/180] [ 8138.207170] LNet: Added LNI 192.168.203.122@tcp125 [8/256/0/180] [ 8138.487791] LNet: Added LNI 192.168.203.122@tcp126 [8/256/0/180] [ 8138.758267] LNet: Added LNI 192.168.203.122@tcp127 [8/256/0/180] [ 8139.027199] LNet: Added LNI 192.168.203.122@tcp128 [8/256/0/180] [ 8139.299534] LNet: Added LNI 192.168.203.122@tcp129 [8/256/0/180] [ 8139.577132] LNet: Added LNI 192.168.203.122@tcp130 [8/256/0/180] [ 8139.840969] LNet: Added LNI 192.168.203.122@tcp131 [8/256/0/180] [ 8140.106243] LNet: Added LNI 192.168.203.122@tcp132 [8/256/0/180] [ 8140.371022] LNet: Added LNI 192.168.203.122@tcp133 [8/256/0/180] [ 8140.632385] LNet: Added LNI 192.168.203.122@tcp134 [8/256/0/180] [ 8140.894849] LNet: Added LNI 192.168.203.122@tcp135 [8/256/0/180] [ 8143.014217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8143.036513] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8143.041172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8144.080121] Lustre: 471350:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.122@tcp: can't add NID 192.168.203.122@tcp130: rc = -75 [ 8144.141193] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8144.152493] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8144.198827] Lustre: lustre-MDT0000: new disk, initializing [ 8144.230020] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8144.235879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8145.520746] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8149.501289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8149.528083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8149.565823] Lustre: 471369:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8149.589229] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8149.590989] Lustre: Skipped 1 previous similar message [ 8149.628369] Lustre: lustre-MDT0001: new disk, initializing [ 8149.640991] Lustre: 472151:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.122@tcp: can't add NID 192.168.203.122@tcp130: rc = -75 [ 8149.643883] Lustre: 472151:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 8149.680951] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8149.692269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8149.695277] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8150.869982] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8151.601499] Lustre: 472499:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.122@tcp: can't add NID 192.168.203.122@tcp130: rc = -75 [ 8151.604243] Lustre: 472499:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 8153.891389] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8155.018520] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8157.233348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8157.252912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8157.369517] Lustre: lustre-OST0000: new disk, initializing [ 8157.371548] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8157.402721] Lustre: 473121:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.122@tcp: can't add NID 192.168.203.122@tcp130: rc = -75 [ 8157.405144] Lustre: 473121:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [ 8157.420363] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8159.136599] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8159.502238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8159.507726] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8159.563824] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8162.202527] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8164.832664] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8164.833449] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8164.835244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8164.836159] LustreError: Skipped 1 previous similar message [ 8164.839723] Lustre: Skipped 1 previous similar message [ 8169.954587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8169.956180] Lustre: Skipped 2 previous similar messages [ 8170.606236] LustreError: 473955:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8170.640259] Lustre: server umount lustre-OST0000 complete [ 8175.072700] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8175.073363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8175.077624] Lustre: Skipped 1 previous similar message [ 8180.192429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8180.194807] Lustre: Skipped 2 previous similar messages [ 8185.312028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8185.313771] Lustre: Skipped 1 previous similar message [ 8186.335099] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8186.365359] LustreError: 474157:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8186.367067] LustreError: 474157:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8186.394672] Lustre: server umount lustre-MDT0000 complete [ 8187.590280] LustreError: 473115:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763161438 with bad export cookie 12805927248507192249 [ 8187.593621] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8187.595685] LustreError: 473115:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8187.661490] LustreError: 474361:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8187.663145] LustreError: 474361:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8187.708355] Lustre: server umount lustre-MDT0001 complete [ 8190.946388] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 8191.866344] Key type lgssc unregistered [ 8192.008395] LNet: 474900:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8192.012560] LNetError: 474900:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8192.015601] LNet: Removed LNI 192.168.203.122@tcp [ 8193.055106] LNet: Removed LNI 192.168.203.122@tcp100 [ 8193.057366] LNetError: 474900:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8193.061295] LNetError: 474900:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 8193.065594] LNet: Removed LNI 192.168.203.122@tcp101 [ 8194.079125] LNet: Removed LNI 192.168.203.122@tcp102 [ 8194.081558] LNetError: 474900:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8194.085578] LNetError: 474900:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 8194.089968] LNet: Removed LNI 192.168.203.122@tcp103 [ 8195.103152] LNet: Removed LNI 192.168.203.122@tcp104 [ 8195.105286] LNet: Removed LNI 192.168.203.122@tcp105 [ 8195.107369] LNet: Removed LNI 192.168.203.122@tcp106 [ 8195.109471] LNet: Removed LNI 192.168.203.122@tcp107 [ 8195.111590] LNet: Removed LNI 192.168.203.122@tcp108 [ 8195.113713] LNet: Removed LNI 192.168.203.122@tcp109 [ 8195.115785] LNet: Removed LNI 192.168.203.122@tcp110 [ 8195.117900] LNet: Removed LNI 192.168.203.122@tcp111 [ 8195.119962] LNet: Removed LNI 192.168.203.122@tcp112 [ 8195.122036] LNet: Removed LNI 192.168.203.122@tcp113 [ 8195.124068] LNet: Removed LNI 192.168.203.122@tcp114 [ 8195.126097] LNet: Removed LNI 192.168.203.122@tcp115 [ 8195.128153] LNet: Removed LNI 192.168.203.122@tcp116 [ 8195.130218] LNet: Removed LNI 192.168.203.122@tcp117 [ 8195.132305] LNet: Removed LNI 192.168.203.122@tcp118 [ 8195.134354] LNet: Removed LNI 192.168.203.122@tcp119 [ 8195.136467] LNet: Removed LNI 192.168.203.122@tcp120 [ 8195.138573] LNet: Removed LNI 192.168.203.122@tcp121 [ 8195.140694] LNet: Removed LNI 192.168.203.122@tcp122 [ 8195.142821] LNet: Removed LNI 192.168.203.122@tcp123 [ 8195.144931] LNet: Removed LNI 192.168.203.122@tcp124 [ 8195.147037] LNet: Removed LNI 192.168.203.122@tcp125 [ 8195.149116] LNet: Removed LNI 192.168.203.122@tcp126 [ 8195.151221] LNet: Removed LNI 192.168.203.122@tcp127 [ 8195.153320] LNet: Removed LNI 192.168.203.122@tcp128 [ 8195.155417] LNet: Removed LNI 192.168.203.122@tcp129 [ 8195.157179] LNet: Removed LNI 192.168.203.122@tcp130 [ 8195.158287] LNet: Removed LNI 192.168.203.122@tcp131 [ 8195.159357] LNet: Removed LNI 192.168.203.122@tcp132 [ 8195.160377] LNet: Removed LNI 192.168.203.122@tcp133 [ 8195.161402] LNet: Removed LNI 192.168.203.122@tcp134 [ 8195.171388] LNet: Removed LNI 192.168.203.122@tcp135 [ 8195.407371] Key type .llcrypt unregistered [ 8195.408496] Key type ._llcrypt unregistered [ 8202.267100] Key type ._llcrypt registered [ 8202.268060] Key type .llcrypt registered [ 8202.300738] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 8207.941458] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 8208.303089] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8208.330900] alg: No test for adler32 (adler32-zlib) [ 8209.169281] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 8209.241404] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 8210.815161] Key type lgssc registered [ 8211.176069] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8213.874891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8215.766615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8217.666116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8219.678762] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8221.762943] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 18:04:31 (1763161471) [ 8227.320108] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 8228.288989] Key type lgssc unregistered [ 8228.414319] LNet: 478911:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8228.416324] LNetError: 478911:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8228.424405] LNet: Removed LNI 192.168.203.122@tcp [ 8228.713477] Key type .llcrypt unregistered [ 8228.714512] Key type ._llcrypt unregistered [ 8235.051258] Key type ._llcrypt registered [ 8235.052622] Key type .llcrypt registered [ 8235.091070] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 8235.586233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8235.593457] alg: No test for adler32 (adler32-zlib) [ 8236.449266] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 8236.536525] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 8238.119156] Key type lgssc registered [ 8238.495836] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8241.566972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8241.599111] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8241.603319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8242.689497] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8242.699581] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8242.732721] Lustre: lustre-MDT0000: new disk, initializing [ 8242.755948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8242.762192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8244.003985] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8248.090383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8248.112990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8248.144066] Lustre: 480081:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8248.167290] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8248.168759] Lustre: Skipped 1 previous similar message [ 8248.201383] Lustre: lustre-MDT0001: new disk, initializing [ 8248.233199] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8248.247179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8248.250076] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8249.472293] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8252.454866] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8253.569630] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8255.720781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8255.748845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8255.866179] Lustre: lustre-OST0000: new disk, initializing [ 8255.867976] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8255.899442] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8257.664618] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8260.637315] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8261.124113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8261.129265] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8261.170652] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8261.699386] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing lnet_if_list [ 8263.210573] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 8263.221674] LNet: Added LNI 192.168.203.122@tcp10 [8/256/0/180] [ 8264.880211] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 8264.890302] LNet: Added LNI 192.168.203.122@tcp11 [8/256/0/180] [ 8266.108720] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 8266.118917] LNet: Added LNI 192.168.203.122@tcp12 [8/256/0/180] [ 8267.401261] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 8267.410895] LNet: Added LNI 192.168.203.122@tcp13 [8/256/0/180] [ 8268.696523] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 8268.706716] LNet: Added LNI 192.168.203.122@tcp14 [8/256/0/180] [ 8269.981149] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 8269.990485] LNet: Added LNI 192.168.203.122@tcp15 [8/256/0/180] [ 8271.313612] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 8271.322619] LNet: Added LNI 192.168.203.122@tcp16 [8/256/0/180] [ 8272.746272] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 8272.755602] LNet: Added LNI 192.168.203.122@tcp17 [8/256/0/180] [ 8274.186525] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 8274.196636] LNet: Added LNI 192.168.203.122@tcp18 [8/256/0/180] [ 8275.681826] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 8275.692883] LNet: Added LNI 192.168.203.122@tcp19 [8/256/0/180] [ 8277.200202] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 8277.211814] LNet: Added LNI 192.168.203.122@tcp20 [8/256/0/180] [ 8278.842624] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 8278.854115] LNet: Added LNI 192.168.203.122@tcp21 [8/256/0/180] [ 8280.404323] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 8280.413991] LNet: Added LNI 192.168.203.122@tcp22 [8/256/0/180] [ 8281.997581] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 8282.007100] LNet: Added LNI 192.168.203.122@tcp23 [8/256/0/180] [ 8283.607232] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 8283.616418] LNet: Added LNI 192.168.203.122@tcp24 [8/256/0/180] [ 8285.219535] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 8285.228681] LNet: Added LNI 192.168.203.122@tcp25 [8/256/0/180] [ 8286.913209] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 8286.923546] LNet: Added LNI 192.168.203.122@tcp26 [8/256/0/180] [ 8288.693638] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 8288.703552] LNet: Added LNI 192.168.203.122@tcp27 [8/256/0/180] [ 8288.713357] LNetError: 479341:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.122 [ 8288.718300] LNetError: Protocol error connecting to 192.168.203.122@tcp26 at host 192.168.203.122:988: is it running a compatible version of Lustre? [ 8290.515870] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 8290.527952] LNet: Added LNI 192.168.203.122@tcp28 [8/256/0/180] [ 8290.532399] LNetError: 479344:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.122 [ 8290.535428] LNetError: Protocol error connecting to 192.168.203.122@tcp26 at host 192.168.203.122:988: is it running a compatible version of Lustre? [ 8290.538267] LNetError: Skipped 1 previous similar message [ 8292.337160] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 8292.348033] LNet: Added LNI 192.168.203.122@tcp29 [8/256/0/180] [ 8292.353676] LNetError: 479341:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.122 [ 8292.359331] LNetError: 479341:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 1 previous similar message [ 8292.363549] LNetError: Protocol error connecting to 192.168.203.122@tcp27 at host 192.168.203.122:988: is it running a compatible version of Lustre? [ 8294.208415] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 8294.218931] LNet: Added LNI 192.168.203.122@tcp30 [8/256/0/180] [ 8294.944131] LNet: 1 local NIs in recovery (showing 1): 192.168.203.122@tcp29 [ 8295.968376] LNetError: 479344:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.122 [ 8295.970924] LNetError: 479344:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 1 previous similar message [ 8295.972780] LNetError: Protocol error connecting to 192.168.203.122@tcp29 at host 192.168.203.122:988: is it running a compatible version of Lustre? [ 8295.975163] LNetError: Skipped 1 previous similar message [ 8296.283626] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 8296.294585] LNet: Added LNI 192.168.203.122@tcp31 [8/256/0/180] [ 8298.396665] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 8298.407913] LNet: Added LNI 192.168.203.122@tcp32 [8/256/0/180] [ 8300.369197] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 8300.378399] LNet: Added LNI 192.168.203.122@tcp33 [8/256/0/180] [ 8301.088323] LNetError: 479343:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.122 [ 8301.091262] LNetError: 479343:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 6 previous similar messages [ 8301.093446] LNetError: Protocol error connecting to 192.168.203.122@tcp32 at host 192.168.203.122:988: is it running a compatible version of Lustre? [ 8301.096423] LNetError: Skipped 6 previous similar messages [ 8302.324837] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 8302.338393] LNet: Added LNI 192.168.203.122@tcp34 [8/256/0/180] [ 8304.431848] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 8304.440826] LNet: Added LNI 192.168.203.122@tcp35 [8/256/0/180] [ 8306.408279] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 8306.417136] LNet: Added LNI 192.168.203.122@tcp36 [8/256/0/180] [ 8308.397543] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 8308.410257] LNet: Added LNI 192.168.203.122@tcp37 [8/256/0/180] [ 8309.282585] LNetError: 479342:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.122 [ 8309.285653] LNetError: 479342:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 11 previous similar messages [ 8309.287828] LNetError: Protocol error connecting to 192.168.203.122@tcp33 at host 192.168.203.122:988: is it running a compatible version of Lustre? [ 8309.291033] LNetError: Skipped 11 previous similar messages [ 8310.406763] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 8310.415373] LNet: Added LNI 192.168.203.122@tcp38 [8/256/0/180] [ 8312.445883] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 8312.454705] LNet: Added LNI 192.168.203.122@tcp39 [8/256/0/180] [ 8314.455542] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 8314.464778] LNet: Added LNI 192.168.203.122@tcp40 [8/256/0/180] [ 8316.444257] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 8316.453933] LNet: Added LNI 192.168.203.122@tcp41 [8/256/0/180] [ 8318.534987] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 8318.546186] LNet: Added LNI 192.168.203.122@tcp42 [8/256/0/180] [ 8320.592517] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 8320.604614] LNet: Added LNI 192.168.203.122@tcp43 [8/256/0/180] [ 8322.691847] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 8322.704995] LNet: Added LNI 192.168.203.122@tcp44 [8/256/0/180] [ 8324.984651] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 8324.997579] LNet: Added LNI 192.168.203.122@tcp45 [8/256/0/180] [ 8325.665477] LNetError: 479341:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.122 [ 8325.668274] LNetError: 479341:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 29 previous similar messages [ 8325.673329] LNetError: Protocol error connecting to 192.168.203.122@tcp42 at host 192.168.203.122:988: is it running a compatible version of Lustre? [ 8325.675856] LNetError: Skipped 30 previous similar messages [ 8327.425345] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 8327.433841] LNet: Added LNI 192.168.203.122@tcp46 [8/256/0/180] [ 8329.631794] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 8329.640502] LNet: Added LNI 192.168.203.122@tcp47 [8/256/0/180] [ 8332.018786] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 8332.031215] LNet: Added LNI 192.168.203.122@tcp48 [8/256/0/180] [ 8334.674826] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 8334.685095] LNet: Added LNI 192.168.203.122@tcp49 [8/256/0/180] [ 8337.156649] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 8337.165748] LNet: Added LNI 192.168.203.122@tcp50 [8/256/0/180] [ 8348.130910] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8348.132694] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8348.133380] LustreError: Skipped 1 previous similar message [ 8348.137676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8348.139429] Lustre: Skipped 1 previous similar message [ 8350.188189] LustreError: 504256:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8350.223834] Lustre: server umount lustre-OST0000 complete [ 8353.249502] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8353.250479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8353.252894] Lustre: Skipped 2 previous similar messages [ 8355.359131] LNet: 10 local NIs in recovery (showing 10): 192.168.203.122@tcp48, 192.168.203.122@tcp42, 192.168.203.122@tcp38, 192.168.203.122@tcp40, 192.168.203.122@tcp44, 192.168.203.122@tcp43, 192.168.203.122@tcp35, 192.168.203.122@tcp33, 192.168.203.122@tcp32, 192.168.203.122@tcp29 [ 8357.506757] LustreError: 504457:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8357.508651] LustreError: 504457:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8357.538806] Lustre: server umount lustre-MDT0000 complete [ 8358.433607] LNetError: 479341:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.122 [ 8358.436364] LNetError: 479341:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 46 previous similar messages [ 8358.438445] LNetError: Protocol error connecting to 192.168.203.122@tcp29 at host 192.168.203.122:988: is it running a compatible version of Lustre? [ 8358.441669] LNetError: Skipped 45 previous similar messages [ 8358.728195] LustreError: 480071:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763161609 with bad export cookie 8825147577273163965 [ 8358.731267] LustreError: 480071:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8358.732298] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8358.798834] LustreError: 504659:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8358.801907] LustreError: 504659:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8358.861433] Lustre: server umount lustre-MDT0001 complete [ 8362.162788] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 8363.120673] Key type lgssc unregistered [ 8363.268549] LNet: 505197:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8363.271310] LNetError: 505197:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8363.273610] LNet: Removed LNI 192.168.203.122@tcp [ 8364.319167] LNet: Removed LNI 192.168.203.122@tcp10 [ 8364.320448] LNetError: 505197:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8364.322369] LNetError: 505197:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 8364.324528] LNet: Removed LNI 192.168.203.122@tcp11 [ 8365.343213] LNet: Removed LNI 192.168.203.122@tcp12 [ 8365.344746] LNetError: 505197:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8365.346810] LNetError: 505197:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 8365.348840] LNet: Removed LNI 192.168.203.122@tcp13 [ 8366.367158] LNet: Removed LNI 192.168.203.122@tcp14 [ 8367.391171] LNet: Removed LNI 192.168.203.122@tcp15 [ 8367.393259] LNetError: 505197:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8367.396753] LNetError: 505197:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 8367.400562] LNet: Removed LNI 192.168.203.122@tcp16 [ 8368.415155] LNet: Removed LNI 192.168.203.122@tcp17 [ 8369.439180] LNet: Removed LNI 192.168.203.122@tcp18 [ 8370.463195] LNet: Removed LNI 192.168.203.122@tcp19 [ 8371.487203] LNet: Removed LNI 192.168.203.122@tcp20 [ 8371.488473] LNetError: 505197:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8371.490611] LNetError: 505197:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 8371.492988] LNet: Removed LNI 192.168.203.122@tcp21 [ 8372.511190] LNet: Removed LNI 192.168.203.122@tcp22 [ 8373.535166] LNet: Removed LNI 192.168.203.122@tcp23 [ 8374.559192] LNet: Removed LNI 192.168.203.122@tcp24 [ 8375.583217] LNet: Removed LNI 192.168.203.122@tcp25 [ 8376.607189] LNet: Removed LNI 192.168.203.122@tcp26 [ 8377.631191] LNet: Removed LNI 192.168.203.122@tcp27 [ 8378.655203] LNet: Removed LNI 192.168.203.122@tcp28 [ 8378.657345] LNet: Removed LNI 192.168.203.122@tcp29 [ 8379.679127] LNet: Removed LNI 192.168.203.122@tcp30 [ 8379.680547] LNetError: 505197:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8379.682996] LNetError: 505197:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 8379.686799] LNet: Removed LNI 192.168.203.122@tcp31 [ 8379.688397] LNet: Removed LNI 192.168.203.122@tcp32 [ 8379.689541] LNet: Removed LNI 192.168.203.122@tcp33 [ 8380.703151] LNet: Removed LNI 192.168.203.122@tcp34 [ 8380.705285] LNet: Removed LNI 192.168.203.122@tcp35 [ 8381.727120] LNet: Removed LNI 192.168.203.122@tcp36 [ 8382.751163] LNet: Removed LNI 192.168.203.122@tcp37 [ 8382.752443] LNet: Removed LNI 192.168.203.122@tcp38 [ 8383.775117] LNet: Removed LNI 192.168.203.122@tcp39 [ 8383.776949] LNet: Removed LNI 192.168.203.122@tcp40 [ 8384.799108] LNet: Removed LNI 192.168.203.122@tcp41 [ 8384.800335] LNet: Removed LNI 192.168.203.122@tcp42 [ 8384.801418] LNet: Removed LNI 192.168.203.122@tcp43 [ 8385.823170] LNet: Removed LNI 192.168.203.122@tcp44 [ 8386.847148] LNet: Removed LNI 192.168.203.122@tcp45 [ 8387.871112] LNet: Removed LNI 192.168.203.122@tcp46 [ 8388.895096] LNet: Removed LNI 192.168.203.122@tcp47 [ 8388.896304] LNet: Removed LNI 192.168.203.122@tcp48 [ 8389.919107] LNet: Removed LNI 192.168.203.122@tcp49 [ 8389.927462] LNet: Removed LNI 192.168.203.122@tcp50 [ 8390.200683] Key type .llcrypt unregistered [ 8390.201658] Key type ._llcrypt unregistered [ 8397.046881] Key type ._llcrypt registered [ 8397.047869] Key type .llcrypt registered [ 8397.087491] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 8397.678542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8397.688208] alg: No test for adler32 (adler32-zlib) [ 8398.545600] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 8398.635626] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 8400.223120] Key type lgssc registered [ 8400.608737] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8403.176165] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8403.181420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8404.328223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8405.596096] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8408.329360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8408.453096] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8409.693554] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8411.414100] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8412.633056] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8414.884904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8415.020183] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8416.810297] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8418.524923] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8437.791754] loop0: detected capacity change from 0 to 204800000 [ 8437.850432] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8437.853896] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8437.857135] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8437.876090] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8439.675167] loop0: detected capacity change from 0 to 204800000 [ 8439.730770] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8439.735845] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8439.739195] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8439.786896] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8441.272895] loop0: detected capacity change from 0 to 204800000 [ 8442.935557] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8442.950585] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8442.992648] Lustre: fs15246-OST0000: new disk, initializing [ 8442.994382] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 8443.009384] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8444.639957] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8448.196981] loop1: detected capacity change from 0 to 204800000 [ 8449.874170] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8449.892960] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8449.920246] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 8449.939539] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 8449.973485] Lustre: fs15246-MDT0000: new disk, initializing [ 8450.022595] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8450.030085] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8450.060349] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 8451.104693] Lustre: 510814:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@000000004089dfc4 x1848808989294464/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1763161717 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 8451.166950] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8459.751442] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 8460.191232] LustreError: 510814:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 8460.196118] LustreError: 510814:0:(fid_request.c:255:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [ 8460.200749] LustreError: 510814:0:(osp_precreate.c:459:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [ 8464.190544] LustreError: 511931:0:(obd_class.h:479:obd_check_dev()) Device 30 not setup [ 8464.215866] Lustre: server umount fs15246-MDT0000 complete [ 8464.867820] LustreError: 506417:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8464.875045] LustreError: 506417:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8467.033739] LustreError: 512438:0:(obd_class.h:479:obd_check_dev()) Device 29 not setup [ 8467.036629] LustreError: 512438:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8467.052175] Lustre: server umount fs15246-OST0000 complete [ 8470.495734] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8470.499253] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8470.505294] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8474.591698] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8474.593661] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8474.597068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8476.268096] LustreError: 512938:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8476.269944] LustreError: 512938:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 8476.304729] Lustre: server umount lustre-OST0000 complete [ 8480.225112] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8480.225144] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8480.228578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8480.231604] Lustre: Skipped 3 previous similar messages [ 8483.515574] LustreError: 513140:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8483.517296] LustreError: 513140:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8483.542684] Lustre: server umount lustre-MDT0000 complete [ 8484.684241] LustreError: 507141:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763161735 with bad export cookie 10285911023121031905 [ 8484.687347] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8484.689690] LustreError: 507141:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8484.811143] Lustre: server umount lustre-MDT0001 complete [ 8488.133475] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 8489.014335] Key type lgssc unregistered [ 8489.124374] LNet: 513873:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8489.126245] LNetError: 513873:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8489.133275] LNet: Removed LNI 192.168.203.122@tcp [ 8489.373913] Key type .llcrypt unregistered [ 8489.375498] Key type ._llcrypt unregistered [ 8496.552967] Key type ._llcrypt registered [ 8496.553941] Key type .llcrypt registered [ 8496.585492] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 8496.881839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8496.901049] alg: No test for adler32 (adler32-zlib) [ 8497.737556] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 8497.812480] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 8499.391141] Key type lgssc registered [ 8499.717085] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8501.940789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8503.346788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8504.844669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8506.277412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8510.860079] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 8514.019096] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 8516.940810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8518.894460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8520.780720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8522.837832] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8524.961759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8524.980671] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8524.984611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8526.062833] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8526.072666] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8526.107172] Lustre: lustre-MDT0000: new disk, initializing [ 8526.131347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8526.137544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8527.322412] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8531.222340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8531.243791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8531.272959] Lustre: 518275:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8531.292369] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8531.294646] Lustre: Skipped 1 previous similar message [ 8531.328976] Lustre: lustre-MDT0001: new disk, initializing [ 8531.358259] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8531.368107] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8531.370967] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8532.530748] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8535.445507] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8536.554854] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8538.862269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8538.882753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8538.968652] Lustre: lustre-OST0000: new disk, initializing [ 8538.970475] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8538.994834] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8539.140046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8539.144932] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8539.174883] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8540.635370] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8543.537536] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8544.710058] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8544.775538] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8545.865354] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8545.928202] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8549.345465] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8549.345868] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8549.347501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8549.348560] LustreError: Skipped 1 previous similar message [ 8549.353910] Lustre: Skipped 1 previous similar message [ 8552.554726] LustreError: 521218:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8552.585518] Lustre: server umount lustre-OST0000 complete [ 8554.464584] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8554.465371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8554.467646] Lustre: Skipped 1 previous similar message [ 8554.472358] Lustre: Skipped 2 previous similar messages [ 8559.583938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8559.586441] Lustre: Skipped 1 previous similar message [ 8559.805540] LustreError: 521419:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8559.808506] LustreError: 521419:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8559.838169] Lustre: server umount lustre-MDT0000 complete [ 8560.888104] LustreError: 518266:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763161811 with bad export cookie 1387200705393750565 [ 8560.890475] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8560.890928] LustreError: 518266:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8560.956101] LustreError: 521622:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8560.957784] LustreError: 521622:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8561.005200] Lustre: server umount lustre-MDT0001 complete [ 8567.792196] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 8570.382527] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 8573.249297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8575.423654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8577.423838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8579.331827] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8579.833569] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8309 sec ========== 18:10:29 (1763161829) [ 8580.266710] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 18:10:30 (1763161830) === [ 8581.185619] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 18:10:30 (1763161830) === [ 8592.230713] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 8593.117871] Key type lgssc unregistered [ 8593.244340] LNet: 526469:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8593.246468] LNetError: 526469:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8593.254373] LNet: Removed LNI 192.168.203.122@tcp [ 8593.545346] Key type .llcrypt unregistered [ 8593.546321] Key type ._llcrypt unregistered