[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 552535121 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002532] x2apic enabled [ 0.003018] Switched APIC routing to physical x2apic. [ 0.004019] kvm-guest: setup PV IPIs [ 0.008746] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010011] pid_max: default: 32768 minimum: 301 [ 0.011135] LSM: Security Framework initializing [ 0.012054] Yama: becoming mindful. [ 0.014016] SELinux: Initializing. [ 0.015068] *** VALIDATE selinux *** [ 0.023870] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028728] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029170] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030171] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032106] *** VALIDATE tmpfs *** [ 0.033458] *** VALIDATE proc *** [ 0.034000] *** VALIDATE cgroup *** [ 0.034000] *** VALIDATE cgroup2 *** [ 0.035023] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036132] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038034] Spectre V2 : User space: Vulnerable [ 0.039007] Speculative Store Bypass: Vulnerable [ 0.042012] debug: unmapping init [mem 0xffffffffa1259000-0xffffffffa1260fff] [ 0.044972] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045683] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046021] ... version: 2 [ 0.047015] ... bit width: 48 [ 0.048011] ... generic registers: 4 [ 0.049012] ... value mask: 0000ffffffffffff [ 0.050011] ... max period: 00007fffffffffff [ 0.051014] ... fixed-purpose events: 3 [ 0.052010] ... event mask: 000000070000000f [ 0.053265] rcu: Hierarchical SRCU implementation. [ 0.056426] smp: Bringing up secondary CPUs ... [ 0.057593] x86: Booting SMP configuration: [ 0.058017] .... node #0, CPUs: #1 #2 #3 [ 0.067743] smp: Brought up 1 node, 4 CPUs [ 0.069011] smpboot: Max logical packages: 1 [ 0.070013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.104447] node 0 deferred pages initialised in 29ms [ 0.113172] devtmpfs: initialized [ 0.114000] x86/mm: Memory block size: 128MB [ 0.119047] gcov: version magic: 0x41383552 [ 0.123178] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.129172] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.134636] pinctrl core: initialized pinctrl subsystem [ 0.137351] [ 0.138010] ************************************************************* [ 0.141019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.144015] ** ** [ 0.149017] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.154026] ** ** [ 0.157024] ** This means that this kernel is built to expose internal ** [ 0.161020] ** IOMMU data structures, which may compromise security on ** [ 0.164014] ** your system. ** [ 0.167018] ** ** [ 0.171019] ** If you see this message and you are not debugging the ** [ 0.174020] ** kernel, report this immediately to your vendor! ** [ 0.176012] ** ** [ 0.178011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.181000] ************************************************************* [ 0.185115] NET: Registered protocol family 16 [ 0.186649] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.189060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.193080] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.198534] cpuidle: using governor menu [ 0.200845] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.202646] PCI: Using configuration type 1 for base access [ 0.205136] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.218457] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.220077] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.227160] cryptd: max_cpu_qlen set to 1000 [ 0.234300] ACPI: Added _OSI(Module Device) [ 0.236014] ACPI: Added _OSI(Processor Device) [ 0.237016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.238023] ACPI: Added _OSI(Processor Aggregator Device) [ 0.244408] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.251406] ACPI: Interpreter enabled [ 0.252062] ACPI: PM: (supports S0 S3 S4 S5) [ 0.254013] ACPI: Using IOAPIC for interrupt routing [ 0.255150] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.258474] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.271395] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.273063] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.276021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.281096] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.285504] acpiphp: Slot [2] registered [ 0.287190] acpiphp: Slot [3] registered [ 0.290124] acpiphp: Slot [4] registered [ 0.292116] acpiphp: Slot [5] registered [ 0.293155] acpiphp: Slot [6] registered [ 0.295159] acpiphp: Slot [7] registered [ 0.297201] acpiphp: Slot [8] registered [ 0.299220] acpiphp: Slot [9] registered [ 0.300621] acpiphp: Slot [10] registered [ 0.303166] acpiphp: Slot [11] registered [ 0.304111] acpiphp: Slot [12] registered [ 0.306112] acpiphp: Slot [13] registered [ 0.308090] acpiphp: Slot [14] registered [ 0.310094] acpiphp: Slot [15] registered [ 0.311239] acpiphp: Slot [16] registered [ 0.313179] acpiphp: Slot [17] registered [ 0.315090] acpiphp: Slot [18] registered [ 0.316167] acpiphp: Slot [19] registered [ 0.317166] acpiphp: Slot [20] registered [ 0.319104] acpiphp: Slot [21] registered [ 0.321128] acpiphp: Slot [22] registered [ 0.322155] acpiphp: Slot [23] registered [ 0.324092] acpiphp: Slot [24] registered [ 0.326105] acpiphp: Slot [25] registered [ 0.327098] acpiphp: Slot [26] registered [ 0.328110] acpiphp: Slot [27] registered [ 0.330244] acpiphp: Slot [28] registered [ 0.332099] acpiphp: Slot [29] registered [ 0.333076] acpiphp: Slot [30] registered [ 0.334169] acpiphp: Slot [31] registered [ 0.336069] PCI host bridge to bus 0000:00 [ 0.337019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.340023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.342021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.344032] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.347025] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.352029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.356307] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.361625] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.365824] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.379736] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.388000] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.389019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.390019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.391014] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.393613] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.394800] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.398041] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.400799] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.403881] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.412000] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.416018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.419000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.427018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.436018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.461018] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.478351] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.491024] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.505030] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.536027] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.550000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.560022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.567019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.595025] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.611043] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.616027] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.621018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.637017] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.651489] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.672028] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.687023] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.710018] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.735163] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.751018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.767013] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.775016] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.786814] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.788567] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.790361] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.793761] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.797293] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.802053] iommu: Default domain type: Passthrough [ 0.805545] SCSI subsystem initialized [ 0.807295] ACPI: bus type USB registered [ 0.812220] usbcore: registered new interface driver usbfs [ 0.816096] usbcore: registered new interface driver hub [ 0.819081] usbcore: registered new device driver usb [ 0.821155] pps_core: LinuxPPS API ver. 1 registered [ 0.823009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.825053] PTP clock support registered [ 0.827105] EDAC MC: Ver: 3.0.0 [ 0.830599] PCI: Using ACPI for IRQ routing [ 0.833265] NetLabel: Initializing [ 0.834016] NetLabel: domain hash size = 128 [ 0.836008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.838193] NetLabel: unlabeled traffic allowed by default [ 0.840370] vgaarb: loaded [ 0.844031] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.845007] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.860060] clocksource: Switched to clocksource kvm-clock [ 1.018850] VFS: Disk quotas dquot_6.6.0 [ 1.019995] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.021890] *** VALIDATE ramfs *** [ 1.022889] *** VALIDATE hugetlbfs *** [ 1.024096] pnp: PnP ACPI init [ 1.025967] pnp: PnP ACPI: found 6 devices [ 1.042173] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.044499] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.046055] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.047486] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.049182] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.050877] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 1.052775] NET: Registered protocol family 2 [ 1.054740] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.058744] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.061273] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.065267] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.067766] TCP: Hash tables configured (established 65536 bind 65536) [ 1.069779] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.072050] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.073996] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.076107] NET: Registered protocol family 1 [ 1.078583] RPC: Registered named UNIX socket transport module. [ 1.080246] RPC: Registered udp transport module. [ 1.081388] RPC: Registered tcp transport module. [ 1.082647] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.084634] NET: Registered protocol family 44 [ 1.085939] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.087559] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.089195] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.090776] PCI: CLS 0 bytes, default 64 [ 1.092494] Unpacking initramfs... [ 3.523717] debug: unmapping init [mem 0xffff982a3cc54000-0xffff982a3ffbffff] [ 3.528348] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.531192] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.533357] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.169284] Initialise system trusted keyrings [ 4.170846] Key type blacklist registered [ 4.173147] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.181686] zbud: loaded [ 4.184995] *** VALIDATE nfs *** [ 4.185847] *** VALIDATE nfs4 *** [ 4.186960] pstore: using deflate compression [ 4.189944] Platform Keyring initialized [ 4.361556] NET: Registered protocol family 38 [ 4.363105] Key type asymmetric registered [ 4.367434] Asymmetric key parser 'x509' registered [ 4.369464] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.377230] io scheduler mq-deadline registered [ 4.378949] io scheduler kyber registered [ 4.380375] io scheduler bfq registered [ 4.382695] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.389805] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.399475] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.402665] ACPI: Power Button [PWRF] [ 4.542293] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.678989] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.175018] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 5.385233] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 5.791982] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.852059] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.916874] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.929517] Non-volatile memory driver v1.3 [ 5.934473] Linux agpgart interface v0.103 [ 5.992973] virtio_blk virtio1: [vda] 133016 512-byte logical blocks (68.1 MB/64.9 MiB) [ 5.998925] vda: detected capacity change from 0 to 68104192 [ 6.032800] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 6.040400] vdb: detected capacity change from 0 to 1073741824 [ 6.075961] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.087229] vdc: detected capacity change from 0 to 2621440000 [ 6.117545] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.121511] vdd: detected capacity change from 0 to 2621440000 [ 6.146651] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.153864] vde: detected capacity change from 0 to 4294967296 [ 6.182962] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.193410] vdf: detected capacity change from 0 to 4294967296 [ 6.211874] libphy: Fixed MDIO Bus: probed [ 6.228417] usbcore: registered new interface driver usbserial_generic [ 6.231060] usbserial: USB Serial support registered for generic [ 6.237298] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 6.248049] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 6.255292] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 6.257720] mousedev: PS/2 mouse device common for all mice [ 6.267707] rtc_cmos 00:05: RTC can wake from S4 [ 6.278606] rtc_cmos 00:05: registered as rtc0 [ 6.279924] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 6.288944] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 6.291847] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 6.304240] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 6.307031] intel_pstate: CPU model not supported [ 6.317218] hid: raw HID events driver (C) Jiri Kosina [ 6.337765] usbcore: registered new interface driver usbhid [ 6.341454] usbhid: USB HID core driver [ 6.348016] drop_monitor: Initializing network drop monitor service [ 6.352505] Initializing XFRM netlink socket [ 6.358216] NET: Registered protocol family 10 [ 6.367563] Segment Routing with IPv6 [ 6.368759] NET: Registered protocol family 17 [ 6.384084] mpls_gso: MPLS GSO support [ 6.399373] RAS: Correctable Errors collector initialized. [ 6.404853] AVX version of gcm_enc/dec engaged. [ 6.406154] AES CTR mode by8 optimization enabled [ 6.690996] sched_clock: Marking stable (6690979239, 0)->(7997757791, -1306778552) [ 6.700740] registered taskstats version 1 [ 6.706597] Loading compiled-in X.509 certificates [ 6.712265] zswap: loaded using pool lzo/zbud [ 6.787067] Key type big_key registered [ 6.833331] Key type encrypted registered [ 6.834689] ima: No TPM chip found, activating TPM-bypass! [ 6.848150] ima: Allocated hash algorithm: sha1 [ 6.849629] ima: No architecture policies found [ 6.860732] evm: Initialising EVM extended attributes: [ 6.862142] evm: security.selinux [ 6.866736] evm: security.ima [ 6.870973] evm: security.capability [ 6.871863] evm: HMAC attrs: 0x1 [ 6.880405] rtc_cmos 00:05: setting system clock to 2025-08-05 14:48:36 UTC (1754405316) [ 6.897439] debug: unmapping init [mem 0xffffffffa2203000-0xffffffffa23fffff] [ 6.916740] debug: unmapping init [mem 0xffffffffa0f82000-0xffffffffa1258fff] [ 6.934159] Write protecting the kernel read-only data: 28672k [ 6.948141] debug: unmapping init [mem 0xffffffff9f603000-0xffffffff9f7fffff] [ 6.953021] debug: unmapping init [mem 0xffffffff9ff14000-0xffffffff9fffffff] [ 7.068620] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 7.091144] systemd[1]: Detected virtualization kvm. [ 7.097254] systemd[1]: Detected architecture x86-64. [ 7.098624] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 7.133193] systemd[1]: No hostname configured. [ 7.134566] systemd[1]: Set hostname to . [ 7.141035] random: systemd: uninitialized urandom read (16 bytes read) [ 7.143435] systemd[1]: Initializing machine ID from random generator. [ 7.380059] random: ln: uninitialized urandom read (6 bytes read) [ 7.702043] random: systemd: uninitialized urandom read (16 bytes read) [ 7.709211] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 7.723832] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 7.737684] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 9.797778] device-mapper: uevent: version 1.0.3 [ 9.802817] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 11.850927] virtio_net virtio0 ens2: renamed from eth0 [ 11.987054] random: fast init done [ 12.206499] scsi host0: ata_piix [ 12.759214] scsi host1: ata_piix [ 12.763799] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 12.768822] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 17.391073] random: crng init done [ 17.392658] random: 7 urandom warning(s) missed due to ratelimiting [ 19.921947] dracut-initqueue[591]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 21.275610] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 23.810370] printk: systemd: 26 output lines suppressed due to ratelimiting [ 24.280152] SELinux: Disabled at runtime. [ 24.371336] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 24.382927] systemd[1]: Detected virtualization kvm. [ 24.387160] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 25.548311] systemd[1]: initrd-switch-root.service: Succeeded. [ 25.557752] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 25.570647] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 25.582652] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 25.593770] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 25.621043] systemd[1]: Starting Journal Service... Starting Journal Service... [ 25.632724] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Control Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... [ 26.390757] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target Slices. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 27.568134] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 28.547993] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 28.639777] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 29.251478] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 29.415359] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 35.123209] Key type dns_resolver registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ ***] A start job is running for Configur…only root support (10s / no limit)[ 36.002228] NFS: Registering the id_resolver key type [ 36.003898] Key type id_resolver registered [ 36.009570] Key type id_legacy registered [ **] A start job is running for Configur…only root support (10s / no limit) [ *] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started OpenSSH server daemon. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg333-server login: [ 64.848829] libcfs: loading out-of-tree module taints kernel. [ 64.862421] Key type ._llcrypt registered [ 64.863825] Key type .llcrypt registered [ 64.911524] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 73.232104] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 73.816478] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 73.825639] alg: No test for adler32 (adler32-zlib) [ 74.851196] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 75.194860] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 76.815194] Key type lgssc registered [ 77.443225] Lustre: Echo OBD driver; http://www.lustre.org/ [ 85.094544] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 86.165508] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 90.307782] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 93.481948] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 96.633827] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 103.405000] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 108.652150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 108.685393] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 108.704936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 109.806878] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 109.824525] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 109.866717] Lustre: lustre-MDT0000: new disk, initializing [ 109.911979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 109.920500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 111.467647] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 117.624896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 117.664968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 117.699500] Lustre: 6470:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 117.719841] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 117.722689] Lustre: Skipped 1 previous similar message [ 117.768426] Lustre: lustre-MDT0001: new disk, initializing [ 117.802042] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 117.817922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 117.822281] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 119.407984] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 121.975324] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 125.888946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 125.916708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 126.015465] Lustre: lustre-OST0000: new disk, initializing [ 126.018341] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 126.047299] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 126.512712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 126.517091] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 126.545432] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 128.216798] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 134.199090] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 134.242609] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 134.290389] Lustre: lustre-OST0001: new disk, initializing [ 134.293358] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 134.317599] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 136.654568] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 143.258778] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 144.380136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 144.386046] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 144.403520] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 147.160118] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 154.290785] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing check_logdir /tmp/testlogs/ [ 156.158904] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing yml_node [ 158.155915] Lustre: DEBUG MARKER: Client: 2.16.57.28 [ 159.199322] Lustre: DEBUG MARKER: MDS: 2.16.57.28 [ 160.447603] Lustre: DEBUG MARKER: OSS: 2.16.57.28 [ 161.273098] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Aug 5 10:51:09 EDT 2025 [ 168.930594] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 169.567465] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 184.287852] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 184.291172] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 184.297257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 185.313568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 185.313744] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 185.316987] Lustre: Skipped 2 previous similar messages [ 185.320585] Lustre: Skipped 2 previous similar messages [ 187.651823] LustreError: 12133:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 187.697889] Lustre: server umount lustre-MDT0000 complete [ 189.372830] LustreError: 7397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754405498 with bad export cookie 12860012890069620454 [ 189.374350] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 189.381056] LustreError: 7397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 190.434062] LustreError: 6477:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 190.434709] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 190.444137] LustreError: 6477:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 190.450287] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 190.454763] Lustre: Skipped 2 previous similar messages [ 194.463970] LustreError: 6478:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 194.463989] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 194.469801] LustreError: 6478:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 194.472541] Lustre: Skipped 1 previous similar message [ 195.522488] LustreError: 12335:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 195.526224] LustreError: 12335:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 195.630146] Lustre: server umount lustre-MDT0001 complete [ 197.224253] LustreError: 12536:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 197.226118] LustreError: 12536:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 197.244545] Lustre: server umount lustre-OST0000 complete [ 198.767352] Lustre: server umount lustre-OST0001 complete [ 201.278389] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 205.304205] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 209.511752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 212.527743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 215.224466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 218.011591] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 221.098465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 221.128879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 221.244334] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 221.271596] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 221.318224] Lustre: lustre-MDT0000: new disk, initializing [ 221.370735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 221.381373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 223.033342] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 228.306242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 228.345486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 228.391442] Lustre: 14905:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 228.527286] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 230.163480] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 233.847686] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 235.404229] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 238.339878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 238.369043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 238.500437] Lustre: lustre-OST0000: new disk, initializing [ 238.502772] Lustre: Skipped 1 previous similar message [ 238.505755] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 238.508854] Lustre: Skipped 2 previous similar messages [ 238.550155] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 238.553039] Lustre: Skipped 1 previous similar message [ 239.690806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 239.694755] Lustre: Skipped 1 previous similar message [ 239.697143] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 239.754147] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 240.915940] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 244.368076] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 245.924806] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 246.015267] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 247.529534] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 247.627399] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 250.337368] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 250.339656] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 250.340616] LustreError: Skipped 1 previous similar message [ 250.341554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 250.346513] Lustre: Skipped 2 previous similar messages [ 254.449278] LustreError: 17812:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 254.451827] LustreError: 17812:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 254.530379] Lustre: server umount lustre-OST0000 complete [ 259.554172] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 259.558552] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 259.565819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 259.568059] Lustre: Skipped 1 previous similar message [ 262.371784] LustreError: 18013:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 262.390110] LustreError: 18013:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 262.464162] Lustre: server umount lustre-MDT0000 complete [ 265.562674] LustreError: 16640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754405575 with bad export cookie 12860012890069621959 [ 265.568561] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 265.586072] LustreError: 16640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 265.906287] Lustre: server umount lustre-MDT0001 complete [ 269.856210] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 10:52:57 (1754405577) [ 272.325426] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 274.333685] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 10:53:01 (1754405581) [ 280.118111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 280.597043] LustreError: 18886:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 280.710374] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 284.210539] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 286.185844] LustreError: 18886:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 291.297483] LustreError: 18886:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 292.689119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 296.814716] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 303.136245] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 306.618753] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 311.523566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 314.859109] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 317.826083] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 333.279920] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 333.283075] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 333.288316] Lustre: Skipped 1 previous similar message [ 333.291064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 333.293637] Lustre: Skipped 2 previous similar messages [ 337.970305] LustreError: 21135:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 337.975504] LustreError: 21135:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 338.067199] Lustre: server umount lustre-OST0000 complete [ 354.271134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 354.322684] LustreError: 21338:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 354.325691] LustreError: 21338:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 354.368383] Lustre: server umount lustre-MDT0000 complete [ 356.388899] LustreError: 18866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754405666 with bad export cookie 12860012890069622841 [ 356.392076] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 356.396275] LustreError: 18866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 356.622220] Lustre: server umount lustre-MDT0001 complete [ 362.056867] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 363.656361] Key type lgssc unregistered [ 363.814843] LNet: 22068:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 363.824494] LNetError: 22068:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 363.832930] LNet: Removed LNI 192.168.203.133@tcp [ 364.379433] Key type .llcrypt unregistered [ 364.381112] Key type ._llcrypt unregistered [ 376.995647] Key type ._llcrypt registered [ 376.997078] Key type .llcrypt registered [ 377.046412] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 377.452772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 377.497062] alg: No test for adler32 (adler32-zlib) [ 378.444320] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 378.559267] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 380.167183] Key type lgssc registered [ 380.741621] Lustre: Echo OBD driver; http://www.lustre.org/ [ 384.849858] loop: module loaded [ 384.883206] loop0: detected capacity change from 0 to 204800000 [ 384.977485] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 384.982530] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 385.137706] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 385.145173] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 385.177920] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 385.210534] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 385.231809] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 385.262369] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 387.985110] loop0: detected capacity change from 0 to 204800000 [ 388.063728] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 388.068556] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 388.077953] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 388.155372] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 390.668466] loop0: detected capacity change from 0 to 204800000 [ 393.662972] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 393.691544] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 393.703350] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 394.832386] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 394.862530] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 394.904461] Lustre: lustre-MDT0000: new disk, initializing [ 394.957201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 394.970179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 397.055707] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 401.970841] loop1: detected capacity change from 0 to 204800000 [ 404.824869] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 404.860239] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 405.017800] Lustre: lustre-OST07c6: new disk, initializing [ 405.020662] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 405.023338] Lustre: Skipped 1 previous similar message [ 405.062656] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 407.858869] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 410.650454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 410.654663] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 410.678819] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 502.753168] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 502.755934] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 502.763580] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 507.872515] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 508.399375] LustreError: 26391:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 508.477221] Lustre: server umount lustre-OST07c6 complete [ 512.698774] LustreError: 26891:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 512.701572] LustreError: 26891:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 512.807588] Lustre: server umount lustre-MDT0000 complete [ 519.722242] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 521.091613] Key type lgssc unregistered [ 521.250697] LNet: 27717:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 521.255885] LNetError: 27717:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 521.268915] LNet: Removed LNI 192.168.203.133@tcp [ 521.678753] Key type .llcrypt unregistered [ 521.680466] Key type ._llcrypt unregistered [ 531.336977] Key type ._llcrypt registered [ 531.338451] Key type .llcrypt registered [ 531.396984] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 531.821230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 531.879222] alg: No test for adler32 (adler32-zlib) [ 532.764418] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 532.881584] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 534.503161] Key type lgssc registered [ 534.996952] Lustre: Echo OBD driver; http://www.lustre.org/ [ 538.653449] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 538.665348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 539.897670] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 541.701951] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 546.066982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 546.256112] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 548.075054] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 550.828659] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 552.699254] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 556.045409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 556.225428] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 559.433424] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 562.485758] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 564.456104] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 576.152516] LustreError: 31140:0:(qsd_reint.c:618:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:2 until pending updates are flushed. [ 581.089516] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 581.091981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 581.096770] Lustre: Skipped 1 previous similar message [ 582.771391] LustreError: 31393:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 582.881396] Lustre: server umount lustre-OST0000 complete [ 586.210116] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 586.215359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 586.217599] Lustre: Skipped 1 previous similar message [ 586.222334] Lustre: Skipped 2 previous similar messages [ 590.868133] LustreError: 31593:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 590.871621] LustreError: 31593:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 590.942530] Lustre: server umount lustre-MDT0000 complete [ 591.328056] LustreError: 28941:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 591.336876] LustreError: 28941:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 593.149901] LustreError: 28922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754405902 with bad export cookie 3178534325585728751 [ 593.154237] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 593.157687] LustreError: 28922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 593.221863] LustreError: 31794:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 593.224863] LustreError: 31794:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 593.360356] Lustre: server umount lustre-MDT0001 complete [ 599.234713] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 600.957787] Key type lgssc unregistered [ 601.145445] LNet: 32322:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 601.150449] LNetError: 32322:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 602.213480] LNet: Removed LNI 192.168.203.133@tcp [ 602.704981] Key type .llcrypt unregistered [ 602.706406] Key type ._llcrypt unregistered [ 613.235835] Key type ._llcrypt registered [ 613.237512] Key type .llcrypt registered [ 613.291354] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 613.776339] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 613.823555] alg: No test for adler32 (adler32-zlib) [ 614.756779] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 614.894888] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 616.503196] Key type lgssc registered [ 617.096879] Lustre: Echo OBD driver; http://www.lustre.org/ [ 621.461862] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 621.481116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 622.767240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 625.368802] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 630.575091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 630.831301] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 633.258718] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 636.402273] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 638.667139] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 642.161141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 642.370845] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 645.268933] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 647.531420] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 648.127908] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 663.008582] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 663.012640] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 663.019486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 666.081693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 666.082444] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 669.040086] LustreError: 35790:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 669.134649] Lustre: server umount lustre-OST0000 complete [ 671.201281] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 671.202248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 671.202261] Lustre: Skipped 1 previous similar message [ 671.213622] Lustre: Skipped 1 previous similar message [ 676.321165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 676.323902] Lustre: Skipped 2 previous similar messages [ 677.265617] LustreError: 35993:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 677.270682] LustreError: 35993:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 677.324193] Lustre: server umount lustre-MDT0000 complete [ 679.126641] LustreError: 35994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754405988 with bad export cookie 9146870883193078472 [ 679.130045] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 679.133739] LustreError: 35994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 679.204205] LustreError: 36195:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 679.206519] LustreError: 36195:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 679.314182] Lustre: server umount lustre-MDT0001 complete [ 684.189481] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 685.665258] Key type lgssc unregistered [ 685.851633] LNet: 36721:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 685.855099] LNetError: 36721:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 685.865903] LNet: Removed LNI 192.168.203.133@tcp [ 686.282465] Key type .llcrypt unregistered [ 686.283976] Key type ._llcrypt unregistered [ 698.654617] Key type ._llcrypt registered [ 698.656519] Key type .llcrypt registered [ 698.715028] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 699.143395] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 699.153737] alg: No test for adler32 (adler32-zlib) [ 700.066603] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 700.173827] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 701.783250] Key type lgssc registered [ 702.295475] Lustre: Echo OBD driver; http://www.lustre.org/ [ 706.001562] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 706.021768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 707.302720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 709.172404] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 713.356661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 713.546968] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 715.254302] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 717.572470] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 719.173538] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 722.472921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 722.653080] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 725.248558] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 727.821344] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 731.881180] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 737.249771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 737.251249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 737.255658] Lustre: Skipped 2 previous similar messages [ 737.260461] Lustre: Skipped 2 previous similar messages [ 738.568154] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.33@tcp (stopping) [ 742.087921] LustreError: 40315:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 742.132849] Lustre: server umount lustre-MDT0000 complete [ 742.369293] LustreError: 38091:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 742.376312] LustreError: 38091:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 743.587480] LustreError: 38070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754406053 with bad export cookie 7353805369913137989 [ 743.589251] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 743.591498] LustreError: 38070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 743.676946] LustreError: 38087:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 743.690474] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.33@tcp (stopping) [ 747.488281] LustreError: 38085:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 747.488651] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 747.501721] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 749.698819] LustreError: 40519:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 749.700990] LustreError: 40519:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 749.810660] Lustre: server umount lustre-MDT0001 complete [ 768.174330] LustreError: 40756:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 768.179487] LustreError: 40756:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 768.255533] Lustre: server umount lustre-OST0000 complete [ 774.739966] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 775.972903] Key type lgssc unregistered [ 776.116528] LNet: 41587:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 776.119633] LNetError: 41587:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 776.128676] LNet: Removed LNI 192.168.203.133@tcp [ 776.493345] Key type .llcrypt unregistered [ 776.494766] Key type ._llcrypt unregistered [ 786.461793] Key type ._llcrypt registered [ 786.463323] Key type .llcrypt registered [ 786.515629] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 786.896538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 786.967720] alg: No test for adler32 (adler32-zlib) [ 787.843967] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 787.936877] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 789.527259] Key type lgssc registered [ 789.995128] Lustre: Echo OBD driver; http://www.lustre.org/ [ 793.289777] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 793.300798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 794.517532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 796.124396] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 800.121763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 800.352978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 802.175550] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 804.570856] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 806.095034] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 808.792484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 808.942670] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 811.203492] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 813.380335] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 820.203844] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 830.216458] Lustre: lustre-OST0000: Not available for connect from 192.168.203.33@tcp (stopping) [ 830.433350] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 830.434312] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 830.436851] LustreError: Skipped 1 previous similar message [ 833.707723] LustreError: 45202:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 833.786830] Lustre: server umount lustre-OST0000 complete [ 847.347470] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 847.354753] Lustre: Skipped 1 previous similar message [ 847.357797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 847.360398] Lustre: Skipped 3 previous similar messages [ 850.466190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 850.470132] Lustre: Skipped 1 previous similar message [ 853.518141] LustreError: 45580:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 853.519960] LustreError: 45580:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 853.558669] Lustre: server umount lustre-MDT0000 complete [ 855.308604] LustreError: 42936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754406164 with bad export cookie 13992614720760472786 [ 855.309364] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 855.314040] LustreError: 42936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 861.446386] LustreError: 45782:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 861.449240] LustreError: 45782:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 861.553684] Lustre: server umount lustre-MDT0001 complete [ 865.860018] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 867.106676] Key type lgssc unregistered [ 867.261972] LNet: 46309:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 867.265440] LNetError: 46309:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 867.273383] LNet: Removed LNI 192.168.203.133@tcp [ 867.638840] Key type .llcrypt unregistered [ 867.640392] Key type ._llcrypt unregistered [ 877.952652] Key type ._llcrypt registered [ 877.954137] Key type .llcrypt registered [ 878.000910] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 878.341751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 878.351479] alg: No test for adler32 (adler32-zlib) [ 879.244513] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 879.346763] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 880.943186] Key type lgssc registered [ 881.424809] Lustre: Echo OBD driver; http://www.lustre.org/ [ 884.855209] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 884.864586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 886.080048] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 887.648106] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 891.196348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 891.396707] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 893.116335] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 895.525388] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 897.087465] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 899.846653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 899.985841] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 902.297745] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 904.433784] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 911.273967] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 917.530970] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 918.941444] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 929.803714] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2025-08-0511h03m58s [ 930.400492] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 931.810394] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 931.812220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 931.814770] Lustre: Skipped 1 previous similar message [ 932.834205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 932.836640] Lustre: Skipped 4 previous similar messages [ 935.684202] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.33@tcp (stopping) [ 937.345592] LustreError: 50245:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 937.383038] Lustre: server umount lustre-MDT0000 complete [ 937.953668] LustreError: 47671:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 937.959657] LustreError: 47671:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 938.694138] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 938.977763] LustreError: 48382:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 938.983081] LustreError: 48382:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 941.424274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 941.491416] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 941.608362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 941.638909] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 942.629460] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 943.329876] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 944.756580] Lustre: DEBUG MARKER: Wait for df (36016) ... [ 945.373498] Lustre: DEBUG MARKER: done [ 946.656680] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 946.659897] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 946.663912] Lustre: Skipped 1 previous similar message [ 946.666608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 946.669321] Lustre: Skipped 1 previous similar message [ 947.168336] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 948.593313] LustreError: 51286:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 948.596309] LustreError: 51286:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 948.676454] Lustre: server umount lustre-OST0000 complete [ 952.316813] LustreError: 51490:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 952.319362] LustreError: 51490:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 952.348491] Lustre: server umount lustre-MDT0000 complete [ 952.801439] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 952.801683] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 952.802692] LustreError: 47671:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 952.816016] Lustre: Skipped 2 previous similar messages [ 953.788256] LustreError: 47655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754406263 with bad export cookie 16123837834889725735 [ 953.791028] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 953.791956] LustreError: 47655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 959.933754] LustreError: 51691:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 959.936768] LustreError: 51691:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 960.040828] Lustre: server umount lustre-MDT0001 complete [ 964.501974] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 965.740174] Key type lgssc unregistered [ 965.883358] LNet: 52217:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 965.886988] LNetError: 52217:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 965.896574] LNet: Removed LNI 192.168.203.133@tcp [ 966.248337] Key type .llcrypt unregistered [ 966.249783] Key type ._llcrypt unregistered [ 979.474235] Key type ._llcrypt registered [ 979.475825] Key type .llcrypt registered [ 979.518531] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 986.507115] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 986.897776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 986.960646] alg: No test for adler32 (adler32-zlib) [ 987.829133] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 987.927191] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 989.511159] Key type lgssc registered [ 989.929949] Lustre: Echo OBD driver; http://www.lustre.org/ [ 993.393097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 995.897371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 998.211717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1000.710295] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1003.520603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1003.544327] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1003.552039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1004.663714] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1004.678741] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1004.722352] Lustre: lustre-MDT0000: new disk, initializing [ 1004.758955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1004.768350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1006.357709] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1010.998544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1011.035580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1011.075411] Lustre: 56128:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1011.104456] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1011.106749] Lustre: Skipped 1 previous similar message [ 1011.148124] Lustre: lustre-MDT0001: new disk, initializing [ 1011.192789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1011.208567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1011.212722] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1012.685518] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1015.925528] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1017.189670] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1019.863610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1019.886634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1019.987506] Lustre: lustre-OST0000: new disk, initializing [ 1019.989827] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1020.020948] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1022.119952] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1025.419907] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1026.898259] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1029.122589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1029.126652] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1029.177155] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1030.069076] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 1031.659778] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1031.752766] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1034.211334] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1034.212732] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1034.214201] LustreError: Skipped 1 previous similar message [ 1034.220170] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1034.223341] Lustre: Skipped 1 previous similar message [ 1038.578646] LustreError: 59057:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1038.655028] Lustre: server umount lustre-OST0000 complete [ 1041.888749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1041.892302] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1041.897386] Lustre: Skipped 1 previous similar message [ 1041.899880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1044.448437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1044.449803] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1046.208188] LustreError: 59258:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1046.211682] LustreError: 59258:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1046.255822] Lustre: server umount lustre-MDT0000 complete [ 1047.720101] LustreError: 57862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754406357 with bad export cookie 17312129957949237402 [ 1047.722572] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1047.725213] LustreError: 57862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1047.794748] LustreError: 59461:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1047.797801] LustreError: 59461:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1047.884981] Lustre: server umount lustre-MDT0001 complete [ 1051.124722] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 11:05:59 (1754406359) [ 1053.937943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1054.215069] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1055.869706] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1059.296355] LustreError: 60131:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1059.303624] LustreError: 60131:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1059.386483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1059.563194] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1061.090651] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1063.348858] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1064.914940] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1067.909825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1068.074253] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1070.442092] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1072.793467] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1085.928693] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2025-08-0511h06m34s [ 1086.658947] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1089.687198] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1090.698379] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1090.715678] LustreError: 60125:0:(mdt_handler.c:2329:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1110.812948] Lustre: lustre-MDT0000: Client 72955437-bfea-4ad6-8f81-f34b6fff83aa (at 192.168.203.33@tcp) reconnecting [ 1130.767142] LustreError: 60125:0:(mdt_handler.c:2329:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1131.662816] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1132.727760] Lustre: DEBUG MARKER: done [ 1135.585349] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1135.588030] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1135.589257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1135.593620] Lustre: Skipped 1 previous similar message [ 1135.596211] Lustre: Skipped 1 previous similar message [ 1140.708661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1140.714705] Lustre: Skipped 2 previous similar messages [ 1140.850509] LustreError: 63136:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1140.854671] LustreError: 63136:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1140.933432] Lustre: server umount lustre-OST0000 complete [ 1145.825900] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1145.833115] Lustre: Skipped 1 previous similar message [ 1148.610205] LustreError: 63337:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1148.613534] LustreError: 63337:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1148.648252] Lustre: server umount lustre-MDT0000 complete [ 1150.222029] LustreError: 60111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754406459 with bad export cookie 17312129957949238270 [ 1150.225595] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1150.226610] LustreError: 60111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1150.400131] Lustre: server umount lustre-MDT0001 complete [ 1154.602469] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 1155.790612] Key type lgssc unregistered [ 1155.932530] LNet: 64066:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1155.936061] LNetError: 64066:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1155.944423] LNet: Removed LNI 192.168.203.133@tcp [ 1156.282776] Key type .llcrypt unregistered [ 1156.284518] Key type ._llcrypt unregistered [ 1169.125851] Key type ._llcrypt registered [ 1169.127232] Key type .llcrypt registered [ 1169.179287] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 1176.391173] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 1176.766351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1176.852947] alg: No test for adler32 (adler32-zlib) [ 1177.726288] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 1177.829109] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 1179.423199] Key type lgssc registered [ 1179.891289] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1183.877441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1186.709228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1189.304196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1192.231711] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1195.167433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1195.195317] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1195.204445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1196.307526] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1196.323227] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1196.365736] Lustre: lustre-MDT0000: new disk, initializing [ 1196.412219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1196.422764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1198.045424] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1203.035109] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1203.064640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1203.113189] Lustre: 67977:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1203.140497] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1203.143480] Lustre: Skipped 1 previous similar message [ 1203.183330] Lustre: lustre-MDT0001: new disk, initializing [ 1203.224361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1203.242552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1203.247772] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1204.889985] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1208.659964] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1210.179203] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1213.091854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1213.131157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1213.252795] Lustre: lustre-OST0000: new disk, initializing [ 1213.255652] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1213.297535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1215.664941] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1219.371173] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1221.176058] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1223.178994] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1223.185034] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1223.227304] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1223.334406] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 1225.062827] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1225.155803] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1228.260233] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1228.260771] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1228.261709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1228.264805] LustreError: Skipped 1 previous similar message [ 1228.270625] Lustre: Skipped 1 previous similar message [ 1232.108093] LustreError: 70897:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1232.186807] Lustre: server umount lustre-OST0000 complete [ 1233.888750] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1233.892236] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1233.898818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1233.901515] Lustre: Skipped 1 previous similar message [ 1238.497102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1238.497403] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1239.683329] LustreError: 71098:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1239.685141] LustreError: 71098:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1239.717807] Lustre: server umount lustre-MDT0000 complete [ 1241.303377] LustreError: 68750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754406550 with bad export cookie 14681638420424126490 [ 1241.306146] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1241.309249] LustreError: 68750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1241.383108] LustreError: 71301:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1241.386492] LustreError: 71301:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1241.480136] Lustre: server umount lustre-MDT0001 complete [ 1244.745326] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 11:09:13 (1754406553) [ 1245.330567] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1245.994082] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 11:09:14 (1754406554) [ 1250.549026] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 1252.207432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1252.436977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1252.844728] Lustre: Failing over lustre-MDT0000 [ 1252.847029] LustreError: 72369:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1252.851726] LustreError: 72369:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1252.855655] LustreError: 72369:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 0, retries 0, failed: rc = -5 [ 1252.921333] LustreError: 72421:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1252.923863] LustreError: 72421:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1253.033207] Lustre: server umount lustre-MDT0000 complete [ 1255.594534] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 11:09:24 (1754406564) [ 1258.163755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1258.326091] LustreError: 72988:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1258.335760] LustreError: 72988:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1258.348635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1259.832739] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1263.142573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1264.814895] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1267.048357] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1268.498899] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1271.255757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1271.403788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1271.406700] Lustre: Skipped 1 previous similar message [ 1273.537321] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1275.584918] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1288.147286] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1293.284119] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1293.286658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1293.289319] Lustre: Skipped 2 previous similar messages [ 1293.293121] Lustre: Skipped 2 previous similar messages [ 1296.262311] LustreError: 75337:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1296.264124] LustreError: 75337:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1296.302315] Lustre: server umount lustre-MDT0000 complete [ 1297.753092] LustreError: 72967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754406607 with bad export cookie 14681638420424127631 [ 1297.756438] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1297.758097] LustreError: 72967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1297.905828] Lustre: server umount lustre-MDT0001 complete [ 1299.184228] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1300.977927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1304.445303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1307.553091] LustreError: 66189:0:(client.c:1385:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d1657ccc x1839628536571776/t0(0) o250->MGC192.168.203.133@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1307.701608] LustreError: 74581:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1307.737701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1307.778913] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1309.362057] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1312.737991] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1312.738066] LustreError: 74581:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1313.098774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1314.342279] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 1314.891821] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1315.359228] Lustre: 66192:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1754406608/real 1754406608] req@00000000c89566e5 x1839628536571392/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1754406624 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1317.169636] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1318.762117] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1319.391737] Lustre: 66190:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1754406613/real 1754406613] req@000000000628758f x1839628536571648/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1754406629 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1323.490114] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1323.493304] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1323.499793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1324.514905] Lustre: 66192:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1754406617/real 1754406617] req@000000003af412af x1839628536571904/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1754406633 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1324.525568] Lustre: 66192:0:(client.c:2453:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1327.168813] LustreError: 77785:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1327.171855] LustreError: 77785:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1327.205067] Lustre: server umount lustre-MDT0000 complete [ 1328.781430] LustreError: 72968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754406638 with bad export cookie 14681638420424129521 [ 1328.784944] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1328.787245] LustreError: 72968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1328.938399] Lustre: server umount lustre-MDT0001 complete [ 1331.298340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1334.925644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1338.784050] LustreError: 66189:0:(client.c:1385:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d190c7da x1839628536592256/t0(0) o250->MGC192.168.203.133@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1338.909843] LustreError: 74580:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1338.918437] LustreError: 74580:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 1338.971895] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1340.643070] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1343.969119] LustreError: 74580:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1343.969259] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1343.979087] Lustre: Skipped 2 previous similar messages [ 1344.143063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1344.283095] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1344.285798] Lustre: Skipped 2 previous similar messages [ 1345.317463] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 1345.320608] Lustre: Skipped 1 previous similar message [ 1345.918483] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1346.335115] Lustre: 66192:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1754406639/real 1754406639] req@000000004f05553e x1839628536591872/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1754406655 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1348.133962] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1349.556656] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1350.431116] Lustre: 66191:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1754406644/real 1754406644] req@00000000b47ebab3 x1839628536592128/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1754406660 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1350.441639] Lustre: 66191:0:(client.c:2453:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1354.720476] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1354.725230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1354.727950] Lustre: Skipped 3 previous similar messages [ 1357.827578] LustreError: 80225:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1357.829542] LustreError: 80225:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1357.865215] Lustre: server umount lustre-MDT0000 complete [ 1359.320212] LustreError: 74593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754406668 with bad export cookie 14681638420424130893 [ 1359.324386] LustreError: 74593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1359.325642] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1359.484298] Lustre: server umount lustre-MDT0001 complete [ 1361.932284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1363.267280] Lustre: DEBUG MARKER: files compared the same [ 1374.397811] Lustre: server umount lustre-OST0000 complete [ 1381.165588] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 1382.394499] Key type lgssc unregistered [ 1382.550475] LNet: 81851:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1382.554193] LNetError: 81851:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1382.562428] LNet: Removed LNI 192.168.203.133@tcp [ 1382.939724] Key type .llcrypt unregistered [ 1382.940973] Key type ._llcrypt unregistered [ 1393.096614] Key type ._llcrypt registered [ 1393.097745] Key type .llcrypt registered [ 1393.143456] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 1393.444692] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1393.466300] alg: No test for adler32 (adler32-zlib) [ 1394.338595] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 1394.433508] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 1396.015219] Key type lgssc registered [ 1396.467946] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1399.711827] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1399.720836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1400.937608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1402.602410] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug +malloc all [ 1406.221174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1406.352749] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1407.848877] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug +malloc all [ 1409.909171] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1411.233019] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1413.840331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1413.969305] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1416.134640] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug +malloc all [ 1418.338956] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1426.282984] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1431.519656] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1431.519784] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1431.522812] LustreError: Skipped 1 previous similar message [ 1431.528986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1431.533450] Lustre: Skipped 1 previous similar message [ 1436.640512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1436.644190] Lustre: Skipped 1 previous similar message [ 1437.612077] LustreError: 85463:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1437.675303] Lustre: server umount lustre-OST0000 complete [ 1441.760542] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1441.760876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1441.765952] Lustre: Skipped 2 previous similar messages [ 1445.244618] LustreError: 85665:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1445.246930] LustreError: 85665:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1445.273788] Lustre: server umount lustre-MDT0000 complete [ 1446.630795] LustreError: 83202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754406756 with bad export cookie 6295828634087541462 [ 1446.632421] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1446.635457] LustreError: 83202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1446.704179] LustreError: 85869:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1446.707098] LustreError: 85869:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1446.792639] Lustre: server umount lustre-MDT0001 complete [ 1450.760062] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 1451.906037] Key type lgssc unregistered [ 1452.055991] LNet: 86394:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1452.059166] LNetError: 86394:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1452.066285] LNet: Removed LNI 192.168.203.133@tcp [ 1452.393107] Key type .llcrypt unregistered [ 1452.394317] Key type ._llcrypt unregistered [ 1462.369715] Key type ._llcrypt registered [ 1462.371105] Key type .llcrypt registered [ 1462.411588] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 1462.788907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1462.827042] alg: No test for adler32 (adler32-zlib) [ 1463.689363] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 1463.779199] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 1465.367195] Key type lgssc registered [ 1465.783505] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1468.721159] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1468.728511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1485.215291] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1491.359263] LustreError: 87734:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.133@tcp: failed processing log, type 4: rc = -110 [ 1521.055247] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1523.241740] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1525.275070] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1528.183993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1528.235148] Lustre: *** cfs_fail_loc=706, val=0*** [ 1528.382899] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1528.409108] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1529.778390] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1532.989837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1533.147837] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1534.628308] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1536.641400] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1538.135607] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1541.603676] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1541.608281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1541.612269] Lustre: Skipped 1 previous similar message [ 1546.722487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1546.724932] Lustre: Skipped 1 previous similar message [ 1551.842431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1551.844838] Lustre: Skipped 1 previous similar message [ 1553.375138] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1553.393125] LustreError: 90040:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 1553.465424] Lustre: server umount lustre-OST0000 complete [ 1556.960649] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1556.961256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1556.964990] Lustre: Skipped 1 previous similar message [ 1556.969258] Lustre: Skipped 1 previous similar message [ 1562.081735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1562.084356] Lustre: Skipped 1 previous similar message [ 1569.247283] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1569.291687] LustreError: 90245:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1569.295089] LustreError: 90245:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1569.329194] Lustre: server umount lustre-MDT0000 complete [ 1570.858298] LustreError: 87743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754406880 with bad export cookie 9212076916347411454 [ 1570.861384] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1570.864217] LustreError: 87743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1570.938305] LustreError: 90449:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1570.941175] LustreError: 90449:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1571.039866] Lustre: server umount lustre-MDT0001 complete [ 1575.129285] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 1576.399042] Key type lgssc unregistered [ 1576.554746] LNet: 90975:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1576.558873] LNetError: 90975:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1576.569887] LNet: Removed LNI 192.168.203.133@tcp [ 1576.951446] Key type .llcrypt unregistered [ 1576.952952] Key type ._llcrypt unregistered [ 1586.857850] Key type ._llcrypt registered [ 1586.859514] Key type .llcrypt registered [ 1586.898732] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 1587.205958] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1587.213790] alg: No test for adler32 (adler32-zlib) [ 1588.083867] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 1588.176492] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 1589.759124] Key type lgssc registered [ 1590.164389] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1592.924190] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1592.930951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1595.314380] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1598.017413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1598.243219] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1599.649221] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1602.673241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1602.834377] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1605.002084] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1607.968357] LustreError: 93009:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1607.976254] LustreError: 93009:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 7 previous similar messages [ 1608.075295] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1608.080809] Lustre: MGS: Client 4a17ed28-d50a-4814-b733-4cce7a1b5702 (at 0@lo) reconnecting [ 1608.177355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1608.179764] LustreError: 93019:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1608.210425] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1609.538622] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1618.405339] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1618.405460] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1618.407127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1618.410249] Lustre: Skipped 1 previous similar message [ 1623.523668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1623.526557] Lustre: Skipped 1 previous similar message [ 1623.916100] LustreError: 94946:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1623.979394] Lustre: server umount lustre-OST0000 complete [ 1628.640798] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1628.641461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1628.644450] Lustre: Skipped 1 previous similar message [ 1628.646621] Lustre: Skipped 1 previous similar message [ 1628.648287] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1631.485408] LustreError: 95149:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1631.488109] LustreError: 95149:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1631.499845] Lustre: server umount lustre-MDT0000 complete [ 1632.974351] LustreError: 95352:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1632.976977] LustreError: 95352:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1633.053903] Lustre: server umount lustre-MDT0001 complete [ 1634.591387] Lustre: server umount MGS complete [ 1638.835576] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 11:15:47 (1754406947) [ 1649.988930] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 1653.689996] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 1657.541106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1660.122690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1662.432815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1664.694221] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1667.318370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1667.347277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1668.766286] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1672.899324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1672.920912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1673.007093] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1673.026473] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1673.065854] Lustre: lustre-MDT0001: new disk, initializing [ 1673.096118] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1673.106821] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1673.109684] Lustre: 100043:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1673.112919] LustreError: 100043:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1674.455730] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1678.918478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1678.950850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1679.058786] Lustre: lustre-OST0000: new disk, initializing [ 1679.061443] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1679.089750] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1681.240666] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1684.447805] Lustre: 101285:0:(client.c:1611:after_reply()) @@@ resending request on EINPROGRESS req@00000000fa839247 x1839628966462720/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1754407010 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1684.515172] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1686.918878] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1686.926147] Lustre: MGS: Client dce73541-dd28-47db-aac7-a564de0b17f9 (at 0@lo) reconnecting [ 1686.941389] Lustre: 99334:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1686.950262] LustreError: 99334:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 1686.962325] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1687.001876] Lustre: lustre-MDT0000: new disk, initializing [ 1687.055035] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1688.095716] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1688.095867] Lustre: 101730:0:(client.c:1611:after_reply()) @@@ resending request on EINPROGRESS req@00000000110980e5 x1839628966468096/t0(0) o700->lustre-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1754407013 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1688.107826] Lustre: 101730:0:(client.c:1611:after_reply()) Skipped 1 previous similar message [ 1688.408963] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1693.155172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1693.159161] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1693.175475] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 1713.631934] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1713.635146] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1713.640374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1718.700149] LustreError: 102262:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 1718.702408] LustreError: 102262:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1718.767746] Lustre: server umount lustre-OST0000 complete [ 1721.312995] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1721.313559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1721.317504] Lustre: Skipped 2 previous similar messages [ 1721.317827] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1721.319932] Lustre: Skipped 4 previous similar messages [ 1721.320930] LustreError: Skipped 1 previous similar message [ 1726.330988] LustreError: 102464:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 1726.333219] LustreError: 102464:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1726.342660] Lustre: server umount lustre-MDT0000 complete [ 1726.432111] LustreError: 100013:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1726.436886] LustreError: 100013:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 1733.762857] Lustre: server umount lustre-MDT0001 complete [ 1735.088679] LustreError: 102868:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1735.090770] LustreError: 102868:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1735.116828] Lustre: server umount MGS complete [ 1739.303451] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 11:17:28 (1754407048) [ 1746.136483] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 1747.340236] Key type lgssc unregistered [ 1747.478688] LNet: 104197:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1747.484162] LNetError: 104197:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1747.493528] LNet: Removed LNI 192.168.203.133@tcp [ 1747.858297] Key type .llcrypt unregistered [ 1747.859912] Key type ._llcrypt unregistered [ 1755.965189] Key type ._llcrypt registered [ 1755.966295] Key type .llcrypt registered [ 1756.010309] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 1756.451597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1756.463401] alg: No test for adler32 (adler32-zlib) [ 1757.340295] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 1757.434473] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 1759.031159] Key type lgssc registered [ 1759.464495] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1762.643936] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1762.653060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1763.831230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1765.412331] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1769.014488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1769.169327] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1770.709415] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1772.891619] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1774.464670] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1777.179308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1777.341699] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1779.506908] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1781.730913] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1788.646438] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 1799.137099] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1799.138560] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1799.139652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1799.140860] LustreError: Skipped 1 previous similar message [ 1799.145181] Lustre: Skipped 1 previous similar message [ 1801.071114] LustreError: 107615:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1801.144876] Lustre: server umount lustre-OST0000 complete [ 1804.257128] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1804.258037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1804.260934] Lustre: Skipped 1 previous similar message [ 1804.262641] Lustre: Skipped 1 previous similar message [ 1808.768808] LustreError: 107816:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1808.771695] LustreError: 107816:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1808.812202] Lustre: server umount lustre-MDT0000 complete [ 1809.376293] LustreError: 106078:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1809.382563] LustreError: 106078:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 1810.242310] LustreError: 105350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754407119 with bad export cookie 2758135084370549551 [ 1810.246025] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1810.246465] LustreError: 105350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1810.315796] LustreError: 108018:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1810.318546] LustreError: 108018:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1810.426237] Lustre: server umount lustre-MDT0001 complete [ 1814.678927] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 1815.828112] Key type lgssc unregistered [ 1815.960291] LNet: 108544:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1815.964264] LNetError: 108544:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1815.974541] LNet: Removed LNI 192.168.203.133@tcp [ 1816.327137] Key type .llcrypt unregistered [ 1816.328654] Key type ._llcrypt unregistered [ 1823.711696] Key type ._llcrypt registered [ 1823.713319] Key type .llcrypt registered [ 1823.755662] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 1824.229990] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1824.243158] alg: No test for adler32 (adler32-zlib) [ 1825.097886] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 1825.189401] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 1826.775233] Key type lgssc registered [ 1827.160660] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1829.516550] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1829.516665] LustreError: 109578:0:(tgt_mount.c:2375:server_fill_super()) cfs_race id 716 sleeping [ 1829.520423] Lustre: Skipped 1 previous similar message [ 1829.520442] LustreError: 109579:0:(tgt_mount.c:2375:server_fill_super()) cfs_fail_race id 716 waking [ 1829.529068] LustreError: 109578:0:(tgt_mount.c:2375:server_fill_super()) cfs_fail_race id 716 awake: rc=4995 [ 1829.534350] LustreError: 109578:0:(tgt_mount.c:2387:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1829.539116] LustreError: 109578:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1829.540535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1830.741492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1833.589949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1833.714977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1835.170240] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1836.907692] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing lsmod [ 1837.989082] LustreError: 110742:0:(tgt_mount.c:2375:server_fill_super()) cfs_race id 716 sleeping [ 1837.993422] LustreError: 110743:0:(tgt_mount.c:2375:server_fill_super()) cfs_fail_race id 716 waking [ 1837.996701] LustreError: 110742:0:(tgt_mount.c:2375:server_fill_super()) cfs_fail_race id 716 awake: rc=4995 [ 1837.999613] LustreError: 110742:0:(tgt_mount.c:2387:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1838.003791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1838.004793] LustreError: 110742:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1838.149923] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1843.169219] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1843.170714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1843.173786] Lustre: Skipped 2 previous similar messages [ 1843.176397] Lustre: Skipped 1 previous similar message [ 1844.193644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1844.196543] Lustre: Skipped 3 previous similar messages [ 1845.311846] LustreError: 110929:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1845.351582] Lustre: server umount lustre-MDT0000 complete [ 1846.770586] LustreError: 109598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754407156 with bad export cookie 11067713949411020648 [ 1846.774067] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1846.774957] LustreError: 109598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1848.287227] LustreError: 110053:0:(client.c:1375:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000001e6face3 x1839629214956544/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1848.298703] LustreError: 111135:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1848.301620] LustreError: 111135:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1848.384625] Lustre: server umount lustre-MDT0001 complete [ 1860.145315] LustreError: 111336:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1860.147756] LustreError: 111336:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1860.218594] Lustre: server umount lustre-OST0000 complete [ 1863.620521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1863.835501] LustreError: 111813:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1863.842448] LustreError: 111813:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1863.862950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1865.429337] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1868.842222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1869.002625] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1870.578735] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1872.829461] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1874.335359] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1876.794086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1878.957906] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1881.031898] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1889.254053] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 1894.370498] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1894.372514] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1894.373498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1894.375941] Lustre: Skipped 1 previous similar message [ 1895.151285] LustreError: 114060:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1895.156919] LustreError: 114060:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1895.240784] Lustre: server umount lustre-OST0000 complete [ 1898.911742] Lustre: server umount lustre-MDT0000 complete [ 1899.489940] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1899.490567] LustreError: 113415:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1899.494044] Lustre: Skipped 1 previous similar message [ 1899.503121] LustreError: 113415:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 1900.406679] LustreError: 111794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754407210 with bad export cookie 11067713949411021299 [ 1900.408086] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1900.412031] LustreError: 111794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1900.472968] LustreError: 114463:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1900.476144] LustreError: 114463:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 1900.579818] Lustre: server umount lustre-MDT0001 complete [ 1905.080279] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 1906.322859] Key type lgssc unregistered [ 1906.462479] LNet: 114989:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1906.465641] LNetError: 114989:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1906.475521] LNet: Removed LNI 192.168.203.133@tcp [ 1906.792222] Key type .llcrypt unregistered [ 1906.793278] Key type ._llcrypt unregistered [ 1917.067713] Key type ._llcrypt registered [ 1917.069182] Key type .llcrypt registered [ 1917.107512] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 1917.538233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1917.545785] alg: No test for adler32 (adler32-zlib) [ 1918.411786] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 1918.508326] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 1920.103182] Key type lgssc registered [ 1920.532832] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1923.481251] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1923.489749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1924.661557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1925.992376] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1929.061367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1929.191298] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1930.666904] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1932.711645] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1934.083787] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1936.513154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1936.636717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1938.507868] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1940.457593] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1949.987307] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 1953.022677] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1953.956882] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1955.297966] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1955.298959] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1955.299827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1955.303372] Lustre: Skipped 1 previous similar message [ 1957.037153] LustreError: 118702:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1957.102985] Lustre: server umount lustre-OST0000 complete [ 1959.904048] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1959.906580] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1959.911733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1959.914239] Lustre: Skipped 1 previous similar message [ 1960.508611] LustreError: 118907:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1960.511065] LustreError: 118907:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1960.537969] Lustre: server umount lustre-MDT0000 complete [ 1961.828895] LustreError: 118555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754407271 with bad export cookie 16069309456413457180 [ 1961.831445] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1961.835368] LustreError: 118555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1961.905482] LustreError: 119109:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1961.907959] LustreError: 119109:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1961.981523] Lustre: server umount lustre-MDT0001 complete [ 1965.818795] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 1966.953077] Key type lgssc unregistered [ 1967.095447] LNet: 119636:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1967.098954] LNetError: 119636:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1967.113353] LNet: Removed LNI 192.168.203.133@tcp [ 1967.439229] Key type .llcrypt unregistered [ 1967.440665] Key type ._llcrypt unregistered [ 1975.039169] Key type ._llcrypt registered [ 1975.040705] Key type .llcrypt registered [ 1975.092854] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 1975.482026] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1975.548471] alg: No test for adler32 (adler32-zlib) [ 1976.418356] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 1976.507426] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 1978.095191] Key type lgssc registered [ 1978.508118] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1981.513072] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1981.524147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1982.710944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1984.082569] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1987.241720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1987.408785] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1988.907857] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1991.012683] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1992.456758] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1995.276333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1995.437856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1997.568200] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 1999.624965] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2007.714542] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 2013.154337] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2013.154745] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2013.156580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2013.160677] Lustre: Skipped 1 previous similar message [ 2018.276595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2018.280497] Lustre: Skipped 2 previous similar messages [ 2019.053410] LustreError: 123064:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2019.116767] Lustre: server umount lustre-OST0000 complete [ 2023.392530] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2023.393044] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2023.393737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2023.400643] Lustre: Skipped 1 previous similar message [ 2026.492224] LustreError: 123266:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2026.494484] LustreError: 123266:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2026.520764] Lustre: server umount lustre-MDT0000 complete [ 2027.771364] LustreError: 122963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754407337 with bad export cookie 7320136270193561634 [ 2027.774465] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2027.775906] LustreError: 122963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2027.839278] LustreError: 123467:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2027.841969] LustreError: 123467:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2027.922209] Lustre: server umount lustre-MDT0001 complete [ 2031.778034] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 2032.954804] Key type lgssc unregistered [ 2033.094414] LNet: 123993:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2033.097828] LNetError: 123993:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2033.104302] LNet: Removed LNI 192.168.203.133@tcp [ 2033.412951] Key type .llcrypt unregistered [ 2033.413880] Key type ._llcrypt unregistered [ 2042.847292] Key type ._llcrypt registered [ 2042.848495] Key type .llcrypt registered [ 2042.886470] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 2043.228048] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2043.292682] alg: No test for adler32 (adler32-zlib) [ 2044.185266] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 2044.274787] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 2045.863284] Key type lgssc registered [ 2046.263024] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2049.116898] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2049.124894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2050.301234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2051.761889] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2054.884076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2055.017111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2056.393171] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2058.395409] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2059.765868] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2062.381067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2062.511603] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2064.657878] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2066.656264] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2074.789814] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 2080.004964] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2081.334905] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2081.337587] Lustre: Skipped 2 previous similar messages [ 2083.013872] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2083.018671] Lustre: Skipped 2 previous similar messages [ 2088.955665] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2088.966613] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2092.985952] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.33@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2092.989463] Lustre: Skipped 2 previous similar messages [ 2095.618682] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.33@tcp 192.168.0.[2,10]@tcp [ 2099.007023] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.33/24@tcp in log lustre-MDT0000 [ 2099.010109] Lustre: Skipped 2 previous similar messages [ 2100.922460] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.33/24@tcp [ 2100.925836] Lustre: Skipped 1 previous similar message [ 2110.945147] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2110.945741] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2110.948165] LustreError: Skipped 1 previous similar message [ 2110.952546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2110.956831] Lustre: Skipped 1 previous similar message [ 2113.388106] LustreError: 129059:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2113.445987] Lustre: server umount lustre-OST0000 complete [ 2116.064722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2116.065376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2116.070030] Lustre: Skipped 2 previous similar messages [ 2116.073978] Lustre: Skipped 1 previous similar message [ 2121.184838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2121.187920] Lustre: Skipped 1 previous similar message [ 2126.304074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2126.305650] Lustre: Skipped 1 previous similar message [ 2128.863110] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2128.893115] LustreError: 129260:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2128.894505] LustreError: 129260:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2128.915952] Lustre: server umount lustre-MDT0000 complete [ 2130.102666] LustreError: 125343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754407439 with bad export cookie 16495418726470053495 [ 2130.105883] LustreError: 125343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2130.106067] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2130.172556] LustreError: 129463:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2130.174929] LustreError: 129463:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2130.260097] Lustre: server umount lustre-MDT0001 complete [ 2133.875141] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 2134.950794] Key type lgssc unregistered [ 2135.099975] LNet: 129989:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2135.102826] LNetError: 129989:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2135.112285] LNet: Removed LNI 192.168.203.133@tcp [ 2135.423859] Key type .llcrypt unregistered [ 2135.425549] Key type ._llcrypt unregistered [ 2145.616729] Key type ._llcrypt registered [ 2145.617856] Key type .llcrypt registered [ 2145.659300] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 2145.978887] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2145.986048] alg: No test for adler32 (adler32-zlib) [ 2146.853238] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 2146.947137] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 2148.535158] Key type lgssc registered [ 2148.917354] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2151.499780] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2151.506101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2152.648463] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2152.652533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2152.658920] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.33/24@tcp [ 2154.004792] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2157.030893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2157.135487] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2157.143156] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.33/24@tcp [ 2157.158481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2158.607172] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2160.672595] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2162.150784] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2164.937496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2165.119453] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2167.309740] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2169.272111] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2178.475722] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 2183.650122] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2183.651675] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2183.652437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2183.654840] Lustre: Skipped 1 previous similar message [ 2185.073101] LustreError: 133803:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2185.160280] Lustre: server umount lustre-OST0000 complete [ 2187.744125] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2187.747744] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2187.752313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2187.754803] Lustre: Skipped 1 previous similar message [ 2188.674342] LustreError: 134005:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2188.677322] LustreError: 134005:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2188.713441] Lustre: server umount lustre-MDT0000 complete [ 2188.769207] LustreError: 131499:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2188.769497] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2188.777917] LustreError: 131499:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 2190.146923] LustreError: 133716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754407499 with bad export cookie 719973377673407213 [ 2190.151061] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2190.152311] LustreError: 133716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2190.223517] LustreError: 134205:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2190.226400] LustreError: 134205:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2190.336209] Lustre: server umount lustre-MDT0001 complete [ 2194.139978] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 2195.363143] Key type lgssc unregistered [ 2195.510563] LNet: 134731:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2195.513228] LNetError: 134731:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2195.519322] LNet: Removed LNI 192.168.203.133@tcp [ 2195.870314] Key type .llcrypt unregistered [ 2195.871695] Key type ._llcrypt unregistered [ 2205.175122] Key type ._llcrypt registered [ 2205.176516] Key type .llcrypt registered [ 2205.215562] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 2211.516704] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 2211.931573] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2211.939605] alg: No test for adler32 (adler32-zlib) [ 2212.792202] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 2212.879433] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 2214.463213] Key type lgssc registered [ 2214.859158] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2218.016222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2220.313614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2222.239525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2224.399620] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2226.774248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2226.795919] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2226.801801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2227.889943] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2227.901936] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2227.939097] Lustre: lustre-MDT0000: new disk, initializing [ 2227.968152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2227.976146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2229.399619] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2233.562884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2233.594259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2233.628521] Lustre: 138063:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2233.652816] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2233.654729] Lustre: Skipped 1 previous similar message [ 2233.690918] Lustre: lustre-MDT0001: new disk, initializing [ 2233.723396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2233.736035] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2233.739542] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2235.138091] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2238.564323] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2240.025284] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2242.683873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2242.709893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2242.815357] Lustre: lustre-OST0000: new disk, initializing [ 2242.818347] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2242.850802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2244.814538] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2247.970687] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2249.318858] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2251.767842] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2251.771277] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2251.799953] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2252.460551] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 2253.829285] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2253.911863] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2256.865858] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2256.867427] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2256.867643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2256.872810] Lustre: Skipped 1 previous similar message [ 2260.654447] LustreError: 140993:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2260.729506] Lustre: server umount lustre-OST0000 complete [ 2261.985268] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2261.985832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2261.988732] Lustre: Skipped 1 previous similar message [ 2261.992336] Lustre: Skipped 2 previous similar messages [ 2267.104304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2267.107046] Lustre: Skipped 1 previous similar message [ 2272.223952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2272.226201] Lustre: Skipped 1 previous similar message [ 2276.319269] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2276.363446] LustreError: 141194:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2276.371742] LustreError: 141194:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2276.402123] Lustre: server umount lustre-MDT0000 complete [ 2277.718488] LustreError: 138838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754407587 with bad export cookie 12649244242153631858 [ 2277.721102] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2277.723319] LustreError: 138838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2277.786020] LustreError: 141397:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2277.788204] LustreError: 141397:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2277.868889] Lustre: server umount lustre-MDT0001 complete [ 2280.747943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2280.931458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2282.232562] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2284.993505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2285.107392] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2286.382493] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2288.204376] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2289.517459] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2291.890441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2293.913806] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2295.738413] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2297.127834] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2306.409161] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 2307.765115] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2307.854944] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2311.129326] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2311.152201] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2311.202811] Lustre: lustre-OST0001: new disk, initializing [ 2311.204560] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2311.236619] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2311.239275] Lustre: Skipped 1 previous similar message [ 2313.052821] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2316.258964] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2317.325639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2317.330465] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2317.365680] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2318.387454] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 2319.664811] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2319.737784] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2320.900782] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff92c350f1b800.ost_server_uuid 50 [ 2321.384353] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff92c350f1b800.ost_server_uuid in FULL state after 0 sec [ 2327.522686] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2327.526149] LustreError: Skipped 1 previous similar message [ 2327.526529] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2327.528523] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2327.532392] Lustre: Skipped 1 previous similar message [ 2327.535454] Lustre: Skipped 1 previous similar message [ 2329.324698] LustreError: 146148:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 2329.326772] LustreError: 146148:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2329.346486] Lustre: server umount lustre-OST0001 complete [ 2331.104120] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2331.106431] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2332.643265] LustreError: 143454:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2332.647874] LustreError: 143454:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 5 previous similar messages [ 2336.687219] LustreError: 146352:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2336.689607] LustreError: 146352:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2336.751553] Lustre: server umount lustre-OST0000 complete [ 2341.343855] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2341.345933] LustreError: Skipped 1 previous similar message [ 2341.347489] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2341.351975] Lustre: Skipped 1 previous similar message [ 2341.353807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2341.356144] Lustre: Skipped 3 previous similar messages [ 2344.086505] Lustre: server umount lustre-MDT0000 complete [ 2345.279860] LustreError: 143477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754407654 with bad export cookie 12649244242153632726 [ 2345.282441] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2345.283108] LustreError: 143477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2345.341948] LustreError: 146755:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 2345.343873] LustreError: 146755:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2345.418964] Lustre: server umount lustre-MDT0001 complete [ 2351.653328] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 2352.662243] Key type lgssc unregistered [ 2352.797777] LNet: 147737:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2352.800970] LNetError: 147737:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2352.807480] LNet: Removed LNI 192.168.203.133@tcp [ 2353.131230] Key type .llcrypt unregistered [ 2353.132907] Key type ._llcrypt unregistered [ 2364.046177] Key type ._llcrypt registered [ 2364.047440] Key type .llcrypt registered [ 2364.086378] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 2370.418167] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 2370.765642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2370.796102] alg: No test for adler32 (adler32-zlib) [ 2371.658899] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 2371.755622] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 2373.343219] Key type lgssc registered [ 2373.716249] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2376.747339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2379.099568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2381.170619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2383.264739] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2385.560717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2385.581251] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2385.586865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2386.668917] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2386.679611] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2386.713749] Lustre: lustre-MDT0000: new disk, initializing [ 2386.739890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2386.746833] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2388.105712] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2392.251310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2392.276430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2392.305106] Lustre: 151645:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2392.328352] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2392.331250] Lustre: Skipped 1 previous similar message [ 2392.367684] Lustre: lustre-MDT0001: new disk, initializing [ 2392.404535] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2392.419501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2392.423292] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2393.764185] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2396.840164] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2398.063559] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2400.382028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2400.408171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2400.505126] Lustre: lustre-OST0000: new disk, initializing [ 2400.506909] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2400.535086] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2400.717628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2400.721981] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2400.754352] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2402.461795] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2405.402055] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2406.658397] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2406.726957] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2408.028735] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2408.104209] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2410.980223] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2410.981697] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2410.982661] LustreError: Skipped 1 previous similar message [ 2410.988616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2410.991201] Lustre: Skipped 1 previous similar message [ 2414.829137] LustreError: 154553:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2414.892858] Lustre: server umount lustre-OST0000 complete [ 2418.144042] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2418.146410] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2418.149832] Lustre: Skipped 1 previous similar message [ 2418.151866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2421.216043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2421.216449] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2422.393161] LustreError: 154754:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2422.395044] LustreError: 154754:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2422.421482] Lustre: server umount lustre-MDT0000 complete [ 2423.599091] LustreError: 151637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754407733 with bad export cookie 5655639269520888908 [ 2423.601167] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2423.603636] LustreError: 151637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2423.665167] LustreError: 154955:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2423.667830] LustreError: 154955:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2423.754495] Lustre: server umount lustre-MDT0001 complete [ 2426.352377] Lustre: DEBUG MARKER: == conf-sanity test 47a: server restart does not lose lru_resize ========================================================== 11:28:55 (1754407735) [ 2431.390911] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 2434.456340] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 2437.868032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2440.084535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2442.221494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2444.412033] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2446.806066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2446.830693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2446.910777] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2446.921536] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2446.957495] Lustre: lustre-MDT0000: new disk, initializing [ 2446.982070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2446.989807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2448.439583] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2452.794277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2452.813429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2452.843574] Lustre: 157927:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2452.866456] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2452.868351] Lustre: Skipped 1 previous similar message [ 2452.904497] Lustre: lustre-MDT0001: new disk, initializing [ 2452.935995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2452.949986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2452.954418] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2454.264634] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2457.409578] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2458.601049] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2460.900750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2460.928680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2461.024811] Lustre: lustre-OST0000: new disk, initializing [ 2461.027339] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2461.057300] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2462.956168] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2465.995197] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2468.855223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2468.859534] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2468.887834] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2473.021262] Lustre: Failing over lustre-OST0000 [ 2473.055111] LustreError: 160350:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2473.057412] LustreError: 160350:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2473.124756] Lustre: server umount lustre-OST0000 complete [ 2485.953615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2486.055481] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2486.058670] Lustre: Skipped 1 previous similar message [ 2486.065109] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2486.071868] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2487.554769] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2487.837713] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2489.059154] Lustre: Failing over lustre-MDT0000 [ 2489.120984] LustreError: 161316:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2489.123706] LustreError: 161316:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2489.149306] Lustre: server umount lustre-MDT0000 complete [ 2490.079146] Lustre: 149865:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1754407783/real 1754407783] req@00000000f5e82d42 x1839629788570240/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1754407799 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2491.362731] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2491.365736] Lustre: Skipped 2 previous similar messages [ 2494.431292] Lustre: 149862:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1754407788/real 1754407788] req@0000000081286b87 x1839629788571520/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1754407804 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2494.440960] Lustre: 149862:0:(client.c:2453:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2496.480513] LustreError: 157937:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2496.485698] LustreError: 157937:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 7 previous similar messages [ 2497.787992] LustreError: 160251:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2499.551211] Lustre: 149864:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1754407793/real 1754407793] req@000000009f8b7045 x1839629788572672/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1754407809 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2501.600854] LustreError: 157937:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2501.607911] LustreError: 157937:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 2502.075591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2502.115970] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2502.121271] Lustre: 157929:0:(mgc_request.c:1876:mgc_process_log()) MGC192.168.203.133@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2502.200611] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2502.248704] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2502.909191] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2503.583935] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2507.238850] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 2507.245673] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2507.249807] Lustre: Skipped 1 previous similar message [ 2507.266154] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2512.353649] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2512.354957] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2512.355914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2512.355918] Lustre: Skipped 1 previous similar message [ 2512.358357] Lustre: Skipped 1 previous similar message [ 2512.361792] LustreError: Skipped 1 previous similar message [ 2514.417125] LustreError: 162195:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2514.418737] LustreError: 162195:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2514.486485] Lustre: server umount lustre-OST0000 complete [ 2517.472640] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2517.473417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2517.476660] Lustre: Skipped 2 previous similar messages [ 2521.814472] Lustre: server umount lustre-MDT0000 complete [ 2522.591968] LustreError: 157938:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2522.596333] LustreError: 157938:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 2523.110399] LustreError: 158702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754407832 with bad export cookie 5655639269520890945 [ 2523.112969] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2523.114085] LustreError: 158702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2523.263394] Lustre: server umount lustre-MDT0001 complete [ 2526.965849] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 2528.103953] Key type lgssc unregistered [ 2528.251378] LNet: 163124:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2528.253818] LNetError: 163124:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2528.262254] LNet: Removed LNI 192.168.203.133@tcp [ 2528.556868] Key type .llcrypt unregistered [ 2528.558228] Key type ._llcrypt unregistered [ 2547.953302] Key type ._llcrypt registered [ 2547.954197] Key type .llcrypt registered [ 2547.986641] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 2548.343470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2548.390128] alg: No test for adler32 (adler32-zlib) [ 2549.229466] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 2549.303025] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 2550.879161] Key type lgssc registered [ 2551.295880] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2554.185208] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2554.196040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2555.378198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2556.751972] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2559.731263] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2561.652952] LustreError: 164644:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2561.656814] LustreError: 164644:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2561.659110] LustreError: 164644:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -5 [ 2561.715138] LustreError: 165160:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2561.795848] Lustre: server umount lustre-MDT0000 complete [ 2566.694803] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 2569.683278] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 2572.809300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2574.879885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2576.804638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2578.735621] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2580.891240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2580.915727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2581.011612] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2581.025686] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2581.072496] Lustre: lustre-MDT0000: new disk, initializing [ 2581.099670] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2581.107262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2582.495739] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2586.578505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2586.597867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2586.633425] Lustre: 167790:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2586.652701] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2586.654285] Lustre: Skipped 1 previous similar message [ 2586.687216] Lustre: lustre-MDT0001: new disk, initializing [ 2586.714915] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2586.727397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2586.730115] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2587.928700] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2591.054651] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2592.375560] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2594.579803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2594.605857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2594.711832] Lustre: lustre-OST0000: new disk, initializing [ 2594.714343] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2594.744549] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2596.667724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2596.671217] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2596.713220] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2596.736147] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2599.863136] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2601.632038] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2602.248160] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 11:31:51 (1754407911) [ 2668.511542] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2668.512072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2668.516807] Lustre: Skipped 1 previous similar message [ 2668.518646] Lustre: Skipped 2 previous similar messages [ 2668.522408] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2673.631574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2673.634559] Lustre: Skipped 3 previous similar messages [ 2674.426948] LustreError: 171278:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2674.428707] LustreError: 171278:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2674.484553] Lustre: server umount lustre-MDT0000 complete [ 2675.632836] LustreError: 167781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754407985 with bad export cookie 11431660192224486817 [ 2675.633978] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2675.635602] LustreError: 167781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2678.751558] LustreError: 167825:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2678.751684] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2678.756588] LustreError: 167825:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 2678.761303] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2680.799444] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2680.799444] LustreError: 167795:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2681.788382] LustreError: 171480:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2681.790673] LustreError: 171480:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2681.858712] Lustre: server umount lustre-MDT0001 complete [ 2689.321598] LustreError: 171681:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2689.323615] LustreError: 171681:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2689.395215] Lustre: server umount lustre-OST0000 complete [ 2692.500316] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 2695.809147] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 2699.644743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2701.976863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2704.043165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2706.210052] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2708.622328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2708.645358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2708.727415] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2708.738614] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2708.778333] Lustre: lustre-MDT0000: new disk, initializing [ 2708.805604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2708.812540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2710.223970] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2714.550937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2714.574334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2714.605146] Lustre: 174002:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2714.627566] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2714.629295] Lustre: Skipped 1 previous similar message [ 2714.668393] Lustre: lustre-MDT0001: new disk, initializing [ 2714.712890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2714.717572] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2716.044767] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2719.235466] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2720.555418] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2722.859241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2722.884704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2722.980043] Lustre: lustre-OST0000: new disk, initializing [ 2722.982631] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2723.011585] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2723.014417] Lustre: Skipped 1 previous similar message [ 2724.945389] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2728.097702] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2729.516158] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2732.024230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2732.027072] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2732.059404] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2732.659462] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 2733.964721] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2734.042031] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2737.122279] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2737.122329] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2737.123770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2737.125865] LustreError: Skipped 1 previous similar message [ 2737.130599] Lustre: Skipped 1 previous similar message [ 2740.782182] LustreError: 176929:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2740.783874] LustreError: 176929:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2740.852130] Lustre: server umount lustre-OST0000 complete [ 2742.240391] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2742.243616] Lustre: Skipped 1 previous similar message [ 2747.360171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2747.362897] Lustre: Skipped 4 previous similar messages [ 2748.274122] Lustre: server umount lustre-MDT0000 complete [ 2750.719081] LustreError: 174777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754408060 with bad export cookie 11431660192224645633 [ 2750.720306] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2750.724233] LustreError: 174777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2750.838112] LustreError: 177332:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2750.840814] LustreError: 177332:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 2751.056431] Lustre: server umount lustre-MDT0001 complete [ 2757.342138] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:34:25 (1754408065) [ 2768.440929] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 2775.507721] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 2782.708800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2787.912391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2792.222995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2796.657864] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2801.354476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2801.415261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2801.584332] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2801.618489] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2801.679806] Lustre: lustre-MDT0000: new disk, initializing [ 2801.749381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2801.765540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2805.408604] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2812.478601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2812.518927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2812.581452] Lustre: 180309:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2812.757991] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2814.918250] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2819.123521] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2820.940776] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2824.370032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2824.407342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2825.948088] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2826.027826] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2827.636300] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2831.541938] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2836.452875] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2836.456241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2836.456287] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2836.477405] Lustre: Skipped 1 previous similar message [ 2840.626095] LustreError: 182835:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2840.627964] LustreError: 182835:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2840.716362] Lustre: server umount lustre-OST0000 complete [ 2843.618035] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2843.620829] LustreError: Skipped 1 previous similar message [ 2846.689462] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2846.694961] Lustre: Skipped 1 previous similar message [ 2848.686205] Lustre: server umount lustre-MDT0000 complete [ 2850.610600] LustreError: 180301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754408160 with bad export cookie 11431660192224646501 [ 2850.614818] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2850.616405] LustreError: 180301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2850.825445] Lustre: server umount lustre-MDT0001 complete [ 2855.114965] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:36:03 (1754408163) [ 2863.147302] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 2867.739985] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 2872.344615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2875.323707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2877.953617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2880.840946] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2884.258733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2884.290353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2884.400344] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2884.419610] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2884.422932] Lustre: Skipped 3 previous similar messages [ 2884.472682] Lustre: lustre-MDT0000: new disk, initializing [ 2884.475104] Lustre: Skipped 2 previous similar messages [ 2884.526051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2884.528441] Lustre: Skipped 2 previous similar messages [ 2884.535223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2884.537714] Lustre: Skipped 2 previous similar messages [ 2886.171827] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2891.521333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2891.564819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2891.610727] Lustre: 186210:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2891.733986] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2893.347690] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2896.827264] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2898.372406] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2901.253657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2901.286029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2902.128638] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2903.787336] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2907.682599] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2912.224877] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2912.228401] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2912.235303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2912.238293] Lustre: Skipped 4 previous similar messages [ 2916.594097] LustreError: 188742:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2916.597623] LustreError: 188742:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2916.671788] Lustre: server umount lustre-OST0000 complete [ 2922.464513] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2922.467927] LustreError: Skipped 1 previous similar message [ 2924.293021] Lustre: server umount lustre-MDT0000 complete [ 2926.726848] LustreError: 188656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754408236 with bad export cookie 11431660192224647586 [ 2926.730770] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2926.732013] LustreError: 188656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2927.042341] Lustre: server umount lustre-MDT0001 complete [ 2932.911179] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 2934.239144] Key type lgssc unregistered [ 2934.398701] LNet: 189671:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2934.403119] LNetError: 189671:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2934.412492] LNet: Removed LNI 192.168.203.133@tcp [ 2934.787676] Key type .llcrypt unregistered [ 2934.789027] Key type ._llcrypt unregistered [ 2946.219835] Key type ._llcrypt registered [ 2946.221289] Key type .llcrypt registered [ 2946.269491] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 2946.737863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2946.762407] alg: No test for adler32 (adler32-zlib) [ 2947.645377] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 2947.754665] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 2949.359242] Key type lgssc registered [ 2949.856248] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2953.434823] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2953.447517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2954.734271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2956.739862] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2961.274284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2961.495735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2963.287775] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2965.597438] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2967.230761] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2970.448634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2970.656707] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2973.245728] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 2975.571227] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2979.887870] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2995.171748] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2995.173861] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2995.181575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2995.187369] Lustre: Skipped 1 previous similar message [ 3000.290112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3000.292746] Lustre: Skipped 1 previous similar message [ 3000.751415] LustreError: 193288:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3000.848493] Lustre: server umount lustre-OST0000 complete [ 3005.409649] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3005.412836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3005.414878] Lustre: Skipped 2 previous similar messages [ 3005.421090] Lustre: Skipped 1 previous similar message [ 3010.528854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3010.531635] Lustre: Skipped 1 previous similar message [ 3015.648414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3015.650733] Lustre: Skipped 1 previous similar message [ 3017.183142] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3017.230722] LustreError: 193491:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3017.232721] LustreError: 193491:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3017.270842] Lustre: server umount lustre-MDT0000 complete [ 3018.967352] LustreError: 191019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754408328 with bad export cookie 5280545109153121037 [ 3018.971129] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3018.973285] LustreError: 191019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3019.051959] LustreError: 193694:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3019.054124] LustreError: 193694:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3019.149198] Lustre: server umount lustre-MDT0001 complete [ 3023.732472] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 3025.027047] Key type lgssc unregistered [ 3025.174429] LNet: 194220:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3025.178779] LNetError: 194220:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3025.189558] LNet: Removed LNI 192.168.203.133@tcp [ 3025.546954] Key type .llcrypt unregistered [ 3025.548665] Key type ._llcrypt unregistered [ 3036.271395] Key type ._llcrypt registered [ 3036.272918] Key type .llcrypt registered [ 3036.322326] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 3036.724135] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3036.786767] alg: No test for adler32 (adler32-zlib) [ 3037.656076] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 3037.753252] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 3039.351161] Key type lgssc registered [ 3039.802220] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3043.355280] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3043.367167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3044.577910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3046.200048] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3049.936574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3050.105162] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3051.768328] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3054.212874] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3055.776528] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3058.716287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3058.873177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3061.383866] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3063.679658] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3070.121845] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 3079.436963] Lustre: lustre-OST0000: Not available for connect from 192.168.203.33@tcp (stopping) [ 3080.674884] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3080.676975] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3080.678747] LustreError: Skipped 1 previous similar message [ 3080.679735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3080.684139] Lustre: Skipped 1 previous similar message [ 3082.861127] LustreError: 197835:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3082.939758] Lustre: server umount lustre-OST0000 complete [ 3085.297487] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff92c350f1a800.ost_server_uuid 50 [ 3086.061297] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 3100.577556] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3100.578567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3100.582633] Lustre: Skipped 1 previous similar message [ 3100.587481] Lustre: Skipped 1 previous similar message [ 3102.601175] LustreError: 198183:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3102.604384] LustreError: 198183:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3102.648581] Lustre: server umount lustre-MDT0000 complete [ 3104.389172] LustreError: 196299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754408414 with bad export cookie 7019781988713407260 [ 3104.391995] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3104.395244] LustreError: 196299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3104.479984] LustreError: 198384:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3104.482553] LustreError: 198384:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3104.582611] Lustre: server umount lustre-MDT0001 complete [ 3108.391802] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 11:40:17 (1754408417) [ 3111.325657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3111.530098] LustreError: 199055:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3111.540125] LustreError: 199055:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3111.565410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3113.170193] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3116.834543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3117.007338] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3118.819869] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3121.301921] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3122.808964] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3125.832947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3125.992349] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3128.554329] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3131.040425] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3134.160214] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3134.203749] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3134.292492] Lustre: lustre-OST0001: new disk, initializing [ 3134.295395] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3135.573472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3135.579380] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3135.626716] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3136.616903] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3136.907037] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3140.761154] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3142.116436] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3142.117176] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3142.117619] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3142.121607] Lustre: Skipped 1 previous similar message [ 3146.504104] Lustre: lustre-OST0000: Not available for connect from 192.168.203.33@tcp (stopping) [ 3146.506669] Lustre: Skipped 1 previous similar message [ 3148.020151] LustreError: 202231:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3148.022976] LustreError: 202231:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3148.043692] Lustre: server umount lustre-OST0000 complete [ 3150.271763] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3150.369723] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3151.619957] LustreError: 200782:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3152.191456] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3152.297422] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3156.735588] LustreError: 200639:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3156.742345] LustreError: 200639:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3162.596306] LustreError: 200639:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3162.603420] LustreError: 200639:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 3166.689221] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3166.691648] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3166.696970] LustreError: Skipped 1 previous similar message [ 3166.704400] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3166.708522] Lustre: Skipped 3 previous similar messages [ 3167.712860] LustreError: 202922:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3167.720545] LustreError: 202922:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 3168.688165] LustreError: 202921:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3168.703273] LustreError: 202921:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3168.792533] Lustre: server umount lustre-OST0001 complete [ 3172.833148] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3172.840038] Lustre: Skipped 2 previous similar messages [ 3176.619094] Lustre: server umount lustre-MDT0000 complete [ 3177.952574] LustreError: 199049:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3177.958696] LustreError: 199049:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 3178.158328] LustreError: 199034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754408487 with bad export cookie 7019781988713408331 [ 3178.161488] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3178.162989] LustreError: 199034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3178.238906] LustreError: 203325:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 3178.242516] LustreError: 203325:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 3178.340314] Lustre: server umount lustre-MDT0001 complete [ 3180.654578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3182.383492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3184.096939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3185.843553] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3189.254030] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 11:41:37 (1754408497) [ 3192.465960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3192.599536] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3192.618833] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3192.716164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3192.720351] Lustre: Skipped 1 previous similar message [ 3194.436907] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3198.193050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3198.244593] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3198.265441] Lustre: 204816:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3200.000248] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3202.539438] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3204.138440] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3206.952188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3207.042675] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3209.477919] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3211.555262] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3213.341749] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3214.584540] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3214.627620] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3217.494371] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3219.921812] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3223.521191] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3223.521362] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3223.524848] LustreError: Skipped 1 previous similar message [ 3223.525523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3223.530295] Lustre: Skipped 1 previous similar message [ 3223.536817] Lustre: Skipped 5 previous similar messages [ 3227.185137] LustreError: 207977:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3227.188098] LustreError: 207977:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 3227.214451] Lustre: server umount lustre-OST0000 complete [ 3228.641469] LustreError: 206433:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3228.648014] LustreError: 206433:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 3238.884439] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3238.887764] LustreError: Skipped 1 previous similar message [ 3244.438313] Lustre: server umount lustre-OST0001 complete [ 3248.033953] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3248.039333] Lustre: Skipped 3 previous similar messages [ 3252.445301] Lustre: server umount lustre-MDT0000 complete [ 3253.939495] LustreError: 207348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754408563 with bad export cookie 7019781988713409717 [ 3253.942514] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3253.944266] LustreError: 207348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3254.106735] Lustre: server umount lustre-MDT0001 complete [ 3256.337436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3257.831380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3259.356887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3261.075716] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3263.747835] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 11:42:52 (1754408572) [ 3270.383126] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 3274.278285] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 3278.817093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3281.720095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3284.183980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3286.785423] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3289.590115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3289.631867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3289.736442] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3289.750742] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3289.792985] Lustre: lustre-MDT0000: new disk, initializing [ 3289.823339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3289.825251] Lustre: Skipped 3 previous similar messages [ 3289.832347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3291.505066] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3296.729402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3296.763737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3296.810818] Lustre: 212396:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3296.844687] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3296.848030] Lustre: Skipped 1 previous similar message [ 3296.887187] Lustre: lustre-MDT0001: new disk, initializing [ 3296.944525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3296.949141] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3298.543814] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3302.072677] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3303.525680] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3306.237074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3306.270121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3306.377770] Lustre: lustre-OST0000: new disk, initializing [ 3306.380396] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3307.200514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3307.204038] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3307.253544] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3308.805287] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3312.408048] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3314.051756] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3314.150954] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3315.801045] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3315.899887] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3317.731689] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3317.733038] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3317.735056] LustreError: Skipped 1 previous similar message [ 3317.741232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3317.744984] Lustre: Skipped 8 previous similar messages [ 3322.730684] LustreError: 215303:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3322.733835] LustreError: 215303:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 3322.824480] Lustre: server umount lustre-OST0000 complete [ 3330.598942] Lustre: server umount lustre-MDT0000 complete [ 3332.318556] LustreError: 212387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754408641 with bad export cookie 7019781988713411236 [ 3332.323531] LustreError: 212387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3332.324173] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3332.506706] Lustre: server umount lustre-MDT0001 complete [ 3336.612909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3336.834082] LustreError: 216185:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3336.843593] LustreError: 216185:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 10 previous similar messages [ 3338.732487] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3342.858849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3344.765089] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3347.137442] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3348.605173] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3351.346728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3353.858940] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3356.271589] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3358.306025] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3362.534703] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 3364.387046] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3364.485785] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3366.886488] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3366.890855] LustreError: Skipped 1 previous similar message [ 3371.445963] Lustre: server umount lustre-OST0000 complete [ 3373.881487] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3375.480820] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3399.839680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3400.003241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3400.008461] Lustre: Skipped 5 previous similar messages [ 3402.529097] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3404.839235] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3405.599217] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3406.897561] LustreError: 220538:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3406.899531] LustreError: 220538:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 3406.967965] Lustre: server umount lustre-OST0000 complete [ 3409.888506] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3409.892131] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3409.898418] Lustre: Skipped 5 previous similar messages [ 3409.901282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3409.904058] Lustre: Skipped 4 previous similar messages [ 3414.684606] Lustre: server umount lustre-MDT0000 complete [ 3416.161799] LustreError: 216166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754408725 with bad export cookie 7019781988713412111 [ 3416.165430] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3416.166385] LustreError: 216166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 3416.342702] Lustre: server umount lustre-MDT0001 complete [ 3419.763269] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 11:45:28 (1754408728) [ 3422.486546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3422.685037] LustreError: 221611:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3422.691608] LustreError: 221611:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 3424.403222] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3427.986413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3429.684127] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3431.869559] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3433.304933] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3435.962386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3438.247123] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3440.327604] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3442.112872] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3448.405952] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 6 sec [ 3450.057308] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3450.163560] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3452.944518] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3452.979698] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3453.045727] Lustre: lustre-OST0001: new disk, initializing [ 3453.047869] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3455.359918] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3459.006171] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3460.666821] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3462.677934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3462.682609] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3462.724448] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3462.840992] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 3464.617785] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3464.719284] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3471.618831] Lustre: server umount lustre-OST0001 complete [ 3474.053412] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3474.157133] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3475.853884] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3475.951789] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3500.123209] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3502.495280] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3504.653566] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3505.328335] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3505.640948] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3505.646105] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to (at 0@lo) [ 3505.646232] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3510.759774] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3510.764702] LustreError: Skipped 3 previous similar messages [ 3512.256544] Lustre: server umount lustre-OST0001 complete [ 3531.060651] Lustre: server umount lustre-OST0000 complete [ 3538.626021] LustreError: 227880:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3538.628307] LustreError: 227880:0:(obd_class.h:479:obd_check_dev()) Skipped 29 previous similar messages [ 3538.666187] Lustre: server umount lustre-MDT0000 complete [ 3540.210945] LustreError: 221591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754408849 with bad export cookie 7019781988713413049 [ 3540.215256] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3540.216474] LustreError: 221591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3546.469312] Lustre: server umount lustre-MDT0001 complete [ 3548.551788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3549.931148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3551.285841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3552.680324] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3555.117294] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 11:47:43 (1754408863) [ 3557.652612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3557.790484] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3557.803465] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3557.881591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3557.883763] Lustre: Skipped 5 previous similar messages [ 3559.457981] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3563.029625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3563.077966] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3563.091558] Lustre: 229571:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3564.770686] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3566.782380] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3568.184032] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3570.731775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3570.822180] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3573.174369] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3575.294808] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3579.655770] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3579.693654] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3582.221996] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3584.528156] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3585.190195] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3586.866258] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3586.962772] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3588.483419] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3588.580912] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3589.910084] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff92c350f18800.ost_server_uuid 50 [ 3590.487122] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff92c350f18800.ost_server_uuid in FULL state after 0 sec [ 3590.873729] Lustre: Permanently deactivating lustre-OST0001 [ 3590.880738] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3591.804560] Lustre: Permanently reactivating lustre-OST0001 [ 3596.775699] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3596.780806] Lustre: Skipped 9 previous similar messages [ 3596.784119] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3596.786691] Lustre: Skipped 15 previous similar messages [ 3598.978765] Lustre: server umount lustre-OST0001 complete [ 3601.890737] LustreError: 232408:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3601.895713] LustreError: 232408:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 25 previous similar messages [ 3606.724228] Lustre: server umount lustre-OST0000 complete [ 3614.307756] Lustre: server umount lustre-MDT0000 complete [ 3615.805318] LustreError: 232091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754408925 with bad export cookie 7019781988713414323 [ 3615.810612] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3615.996190] Lustre: server umount lustre-MDT0001 complete [ 3618.286515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3619.775262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3621.238632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3622.727059] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3625.149812] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 11:48:53 (1754408933) [ 3625.543500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3628.143918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3628.268939] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3628.281829] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3628.285771] Lustre: Skipped 5 previous similar messages [ 3629.933669] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3633.264710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3633.297880] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3633.311249] Lustre: 235674:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3634.980231] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3637.017968] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3638.350473] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3640.928686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3640.957452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3641.037283] Lustre: Permanently deactivating lustre-OST0000 [ 3641.042296] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3643.222580] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3644.720139] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3645.162798] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3647.580930] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3649.942796] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3652.033562] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3653.658424] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3657.919739] Lustre: Permanently reactivating lustre-OST0000 [ 3657.925719] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 3657.928273] Lustre: Skipped 2 previous similar messages [ 3659.764388] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3659.766696] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3659.771173] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.203.133@tcp (at 0@lo) [ 3659.773289] Lustre: Skipped 1 previous similar message [ 3659.787629] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3659.793635] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3666.914596] LustreError: lustre-OST0001-osc-MDT0001: operation ost_destroy to node 0@lo failed: rc = -107 [ 3666.916927] LustreError: Skipped 5 previous similar messages [ 3669.949079] Lustre: server umount lustre-OST0001 complete [ 3677.359182] Lustre: server umount lustre-OST0000 complete [ 3685.011605] Lustre: server umount lustre-MDT0000 complete [ 3686.384280] LustreError: 237318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754408996 with bad export cookie 7019781988713415954 [ 3686.386791] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3686.387608] LustreError: 237318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 3686.534038] Lustre: server umount lustre-MDT0001 complete [ 3690.348837] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 3691.576175] Key type lgssc unregistered [ 3691.708392] LNet: 240047:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3691.711540] LNetError: 240047:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3691.721448] LNet: Removed LNI 192.168.203.133@tcp [ 3692.050274] Key type .llcrypt unregistered [ 3692.051523] Key type ._llcrypt unregistered [ 3701.012840] Key type ._llcrypt registered [ 3701.014178] Key type .llcrypt registered [ 3701.058611] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 3701.520558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3701.586095] alg: No test for adler32 (adler32-zlib) [ 3702.462163] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 3702.556830] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 3704.143191] Key type lgssc registered [ 3704.602404] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3705.982606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3708.576714] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3708.583617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3709.759251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3711.190765] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3714.365052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3714.385883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3714.426162] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3714.440312] Lustre: 241455:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3714.445040] Lustre: Permanently deactivating lustre-MDT0001 [ 3714.452053] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3714.583711] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3716.019255] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3718.001319] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3718.638840] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3718.641518] LustreError: 241497:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3718.645992] LustreError: 241497:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3718.650320] LustreError: 241497:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 1, failed: rc = -108 [ 3719.450442] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3722.091226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3722.235760] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3724.600836] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3725.283564] LustreError: 243075:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3725.286466] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3725.290246] LustreError: 243075:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3726.687106] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3729.421782] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3729.514294] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3731.734898] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3732.910268] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3733.766917] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3734.951139] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3734.951286] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3739.857157] Lustre: Permanently reactivating lustre-MDT0001 [ 3739.863784] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3739.867161] Lustre: Skipped 2 previous similar messages [ 3740.180650] Lustre: lustre-MDT0001: Client 19e4fc0b-f2b1-4379-9b23-4d9c84a6465e (at 192.168.203.33@tcp) reconnecting [ 3742.316647] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3742.991666] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3745.013328] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3745.114023] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3746.778742] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3746.890413] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3747.978228] Lustre: Permanently deactivating lustre-MDT0001 [ 3747.987568] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3747.990806] Lustre: Skipped 2 previous similar messages [ 3751.238490] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3759.075613] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 3759.079351] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3759.088871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3760.608909] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3760.609196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3760.614469] Lustre: Skipped 2 previous similar messages [ 3760.617198] Lustre: Skipped 2 previous similar messages [ 3762.432557] LustreError: 245543:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 3762.460810] Lustre: server umount lustre-MDT0000 complete [ 3763.942749] LustreError: 241447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754409073 with bad export cookie 13325839372955912735 [ 3763.946636] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3763.947090] LustreError: 241447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3764.018618] LustreError: 245745:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 3764.020806] LustreError: 245745:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3764.123290] Lustre: server umount lustre-MDT0001 complete [ 3771.758101] LustreError: 245946:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 3771.760846] LustreError: 245946:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3771.784239] Lustre: server umount lustre-OST0000 complete [ 3776.151713] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 11:51:24 (1754409084) [ 3779.807890] LustreError: 246950:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3779.810410] LustreError: 246950:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3779.869320] Lustre: server umount lustre-OST0001 complete [ 3781.924629] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 3785.327777] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 3789.203841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3791.776994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3794.162954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3796.675075] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3799.192582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3799.217103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3799.309739] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3799.314339] Lustre: Skipped 2 previous similar messages [ 3799.325961] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3799.366990] Lustre: lustre-MDT0000: new disk, initializing [ 3799.403739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3799.412719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3800.937586] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3805.692071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3805.718875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3805.754315] Lustre: 249120:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3805.779387] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3805.782608] Lustre: Skipped 1 previous similar message [ 3805.823441] Lustre: lustre-MDT0001: new disk, initializing [ 3805.871830] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3805.875845] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3807.311361] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3810.722534] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3812.205225] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3814.788332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3814.812731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3814.913664] Lustre: lustre-OST0000: new disk, initializing [ 3814.917176] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3814.955442] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3814.958531] Lustre: Skipped 1 previous similar message [ 3817.133567] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3820.630635] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3824.634052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3824.637074] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3824.667849] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3826.530247] LustreError: 249126:0:(mdt_handler.c:3249:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3831.043575] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3831.067967] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3831.127607] Lustre: lustre-OST0001: new disk, initializing [ 3831.129957] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3831.163170] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3833.295232] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3836.599657] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3836.623084] LustreError: 249126:0:(mdt_handler.c:3249:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3836.632083] LustreError: 249126:0:(mdt_handler.c:3249:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3837.438922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3837.442642] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3837.474526] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3838.268321] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 3846.727226] LustreError: 249126:0:(mdt_handler.c:3249:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3847.650577] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3847.652922] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3847.654541] LustreError: Skipped 1 previous similar message [ 3847.660768] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3847.664532] Lustre: Skipped 2 previous similar messages [ 3852.769820] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3852.772211] Lustre: Skipped 2 previous similar messages [ 3853.547864] LustreError: 252586:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3853.549847] LustreError: 252586:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3853.567106] Lustre: server umount lustre-OST0001 complete [ 3857.890750] LustreError: 250837:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3857.897522] LustreError: 250837:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 3863.012829] LustreError: 250836:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3863.018330] LustreError: 250836:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 3868.130494] LustreError: 250835:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3868.131260] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3868.132386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3868.132392] Lustre: Skipped 1 previous similar message [ 3868.135265] LustreError: 250835:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 3868.139171] Lustre: Skipped 2 previous similar messages [ 3871.788097] LustreError: 252812:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3871.791298] LustreError: 252812:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3871.852435] Lustre: server umount lustre-OST0000 complete [ 3873.249346] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3873.253188] Lustre: Skipped 1 previous similar message [ 3878.368041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3878.369973] Lustre: Skipped 4 previous similar messages [ 3879.383649] Lustre: server umount lustre-MDT0000 complete [ 3880.718224] LustreError: 249893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754409190 with bad export cookie 13325839372955915549 [ 3880.721175] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3880.721669] LustreError: 249893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3895.263150] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3895.295346] LustreError: 253214:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 3895.297580] LustreError: 253214:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 3895.382682] Lustre: server umount lustre-MDT0001 complete [ 3899.351316] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 3900.622158] Key type lgssc unregistered [ 3900.783522] LNet: 253742:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3900.786150] LNetError: 253742:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3900.793609] LNet: Removed LNI 192.168.203.133@tcp [ 3901.165838] Key type .llcrypt unregistered [ 3901.167476] Key type ._llcrypt unregistered [ 3913.055881] Key type ._llcrypt registered [ 3913.057208] Key type .llcrypt registered [ 3913.096589] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 3919.840528] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 3920.222213] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3920.243806] alg: No test for adler32 (adler32-zlib) [ 3921.100894] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 3921.200901] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 3922.791159] Key type lgssc registered [ 3923.197620] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3926.765399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3929.285893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3931.477674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3933.834048] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3936.430841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3936.453623] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3936.459808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3937.548800] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3937.561119] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3937.599721] Lustre: lustre-MDT0000: new disk, initializing [ 3937.630078] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3937.638771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3939.055037] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3943.784117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3943.816990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3943.864435] Lustre: 257650:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3943.897601] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3943.900237] Lustre: Skipped 1 previous similar message [ 3943.943551] Lustre: lustre-MDT0001: new disk, initializing [ 3943.994250] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3944.010548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3944.015428] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3945.689146] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3949.231700] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3950.660623] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3953.134757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3953.162248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3953.270460] Lustre: lustre-OST0000: new disk, initializing [ 3953.273322] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3953.311298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3955.418861] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3958.794147] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3960.354720] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3962.870126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3962.873263] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3962.904556] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3963.520135] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 3964.901034] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3964.979496] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3967.968943] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3967.969822] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3967.972693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3967.975209] Lustre: Skipped 1 previous similar message [ 3971.692068] LustreError: 260577:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3971.748635] Lustre: server umount lustre-OST0000 complete [ 3973.088528] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3973.089291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3973.092887] Lustre: Skipped 1 previous similar message [ 3973.094991] Lustre: Skipped 1 previous similar message [ 3978.207924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3978.210222] Lustre: Skipped 1 previous similar message [ 3979.065596] LustreError: 260778:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3979.067346] LustreError: 260778:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3979.098857] Lustre: server umount lustre-MDT0000 complete [ 3980.339274] LustreError: 257641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754409289 with bad export cookie 14440031344095203867 [ 3980.342135] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3980.342550] LustreError: 257641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3980.399188] LustreError: 260979:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3980.400869] LustreError: 260979:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3980.488425] Lustre: server umount lustre-MDT0001 complete [ 3983.405294] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 11:54:52 (1754409292) [ 3985.587370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3985.749963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3987.056407] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3989.921762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3990.032326] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3991.400731] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 3993.294423] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3994.620224] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3997.127490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3999.093242] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4000.834840] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4051.427196] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4051.428896] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4051.429810] LustreError: Skipped 2 previous similar messages [ 4051.430580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4051.433450] Lustre: Skipped 1 previous similar message [ 4055.020125] LustreError: 263972:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4055.022092] LustreError: 263972:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4055.091170] Lustre: server umount lustre-OST0000 complete [ 4056.609217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 4061.001456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4061.134990] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4061.136834] Lustre: Skipped 1 previous similar message [ 4063.161160] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4065.061923] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4071.395861] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4071.399076] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4071.399147] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.203.133@tcp (at 0@lo) [ 4071.406058] Lustre: Skipped 1 previous similar message [ 4102.116084] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4102.117920] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4102.119366] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4102.119371] Lustre: Skipped 1 previous similar message [ 4102.121922] Lustre: Skipped 1 previous similar message [ 4102.123919] LustreError: Skipped 1 previous similar message [ 4104.559158] LustreError: 265630:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4104.561904] LustreError: 265630:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4104.652825] Lustre: server umount lustre-OST0000 complete [ 4107.232851] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4107.237730] Lustre: Skipped 1 previous similar message [ 4112.151697] Lustre: server umount lustre-MDT0000 complete [ 4112.352779] LustreError: 261650:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4112.356845] LustreError: 261650:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 4113.411105] LustreError: 265531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754409423 with bad export cookie 14440031344095204749 [ 4113.413794] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4113.414674] LustreError: 265531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4113.478151] LustreError: 266035:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4113.480238] LustreError: 266035:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 4113.558585] Lustre: server umount lustre-MDT0001 complete [ 4117.049052] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 4118.198237] Key type lgssc unregistered [ 4118.342380] LNet: 266560:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4118.344536] LNetError: 266560:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4118.356376] LNet: Removed LNI 192.168.203.133@tcp [ 4118.662681] Key type .llcrypt unregistered [ 4118.664126] Key type ._llcrypt unregistered [ 4127.688463] Key type ._llcrypt registered [ 4127.689347] Key type .llcrypt registered [ 4127.728645] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 4128.099843] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4128.146298] alg: No test for adler32 (adler32-zlib) [ 4128.994499] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 4129.075972] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 4130.663238] Key type lgssc registered [ 4131.072920] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4133.824257] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4133.833245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4135.046817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4136.591762] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4139.730482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4139.864102] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4141.270584] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4143.227197] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4144.494935] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4146.897227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4147.011725] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4148.970552] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4150.914868] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4160.362888] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4170.725378] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4170.727112] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4170.728307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4170.729716] LustreError: Skipped 1 previous similar message [ 4170.732765] Lustre: Skipped 1 previous similar message [ 4175.340195] LustreError: 270946:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4175.417522] Lustre: server umount lustre-OST0000 complete [ 4180.961158] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4180.961309] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4180.961966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4180.961971] Lustre: Skipped 1 previous similar message [ 4180.965714] Lustre: Skipped 1 previous similar message [ 4186.080106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4186.081770] Lustre: Skipped 2 previous similar messages [ 4191.199189] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4191.200641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4191.205843] Lustre: Skipped 1 previous similar message [ 4191.237241] LustreError: 271148:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4191.239088] LustreError: 271148:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4191.267578] Lustre: server umount lustre-MDT0000 complete [ 4192.601137] LustreError: 268642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754409502 with bad export cookie 17333810725228862847 [ 4192.604232] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4192.605116] LustreError: 268642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4192.666865] LustreError: 271351:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4192.669405] LustreError: 271351:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4192.749827] Lustre: server umount lustre-MDT0001 complete [ 4196.558737] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 4197.746891] Key type lgssc unregistered [ 4197.883572] LNet: 271877:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4197.886105] LNetError: 271877:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4197.893383] LNet: Removed LNI 192.168.203.133@tcp [ 4198.241993] Key type .llcrypt unregistered [ 4198.243054] Key type ._llcrypt unregistered [ 4205.980390] Key type ._llcrypt registered [ 4205.981835] Key type .llcrypt registered [ 4206.034564] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 4206.536246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4206.547250] alg: No test for adler32 (adler32-zlib) [ 4207.478967] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 4207.620450] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 4209.231278] Key type lgssc registered [ 4210.042224] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4215.334789] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4215.346168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4216.681808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4219.371471] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4225.212127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4225.466510] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4228.202428] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4231.479471] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4233.593802] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4236.840210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4237.002983] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4239.477627] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4241.763790] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4242.154976] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4256.737345] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4256.739436] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4256.743152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4256.744595] Lustre: Skipped 1 previous similar message [ 4257.645833] LustreError: 275443:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4257.715396] Lustre: server umount lustre-OST0000 complete [ 4261.343845] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4261.345555] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4261.349605] Lustre: Skipped 1 previous similar message [ 4261.351822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4265.153992] LustreError: 275646:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4265.156425] LustreError: 275646:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4265.184724] Lustre: server umount lustre-MDT0000 complete [ 4266.555726] LustreError: 275357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754409576 with bad export cookie 5726095854624998102 [ 4266.560145] LustreError: 275357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4266.560662] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4266.638783] LustreError: 275848:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4266.641575] LustreError: 275848:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4266.749803] Lustre: server umount lustre-MDT0001 complete [ 4270.619960] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 4271.847204] Key type lgssc unregistered [ 4271.992633] LNet: 276374:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4271.995439] LNetError: 276374:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4272.003460] LNet: Removed LNI 192.168.203.133@tcp [ 4272.355510] Key type .llcrypt unregistered [ 4272.356426] Key type ._llcrypt unregistered [ 4283.115047] Key type ._llcrypt registered [ 4283.116102] Key type .llcrypt registered [ 4283.164969] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 4283.587255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4283.666314] alg: No test for adler32 (adler32-zlib) [ 4284.528021] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 4284.623344] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 4286.223157] Key type lgssc registered [ 4286.667020] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4289.570391] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4289.579194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4290.753431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4292.258979] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4295.585395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4295.747579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4297.368830] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4299.529374] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4300.949126] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4303.536845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4303.688168] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4305.838546] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4307.882450] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4317.033910] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4327.397264] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4327.397514] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4327.401759] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4327.405124] Lustre: Skipped 1 previous similar message [ 4328.494188] LustreError: 280760:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4328.582602] Lustre: server umount lustre-OST0000 complete [ 4331.488297] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4331.492724] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4331.500242] Lustre: Skipped 1 previous similar message [ 4331.503380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4332.301040] LustreError: 280962:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4332.303394] LustreError: 280962:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4332.359359] Lustre: server umount lustre-MDT0000 complete [ 4332.513508] LustreError: 277742:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4332.514473] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4332.524041] LustreError: 277742:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 4334.687104] LustreError: 277721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754409644 with bad export cookie 14426331886217114298 [ 4334.690294] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4334.698624] LustreError: 277721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4334.810067] LustreError: 281162:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4334.812511] LustreError: 281162:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4334.980920] Lustre: server umount lustre-MDT0001 complete [ 4341.193770] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 4342.905828] Key type lgssc unregistered [ 4343.082709] LNet: 281689:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4343.086781] LNetError: 281689:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4343.096470] LNet: Removed LNI 192.168.203.133@tcp [ 4343.555137] Key type .llcrypt unregistered [ 4343.556645] Key type ._llcrypt unregistered [ 4353.909965] Key type ._llcrypt registered [ 4353.911756] Key type .llcrypt registered [ 4353.990509] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 4354.490527] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4354.546793] alg: No test for adler32 (adler32-zlib) [ 4355.492439] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 4355.642049] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 4357.327256] Key type lgssc registered [ 4357.872972] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4361.802155] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4361.811395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4363.132781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4365.404824] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4369.492426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4369.651771] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4371.420556] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4374.231276] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4375.983454] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4379.463208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4379.654953] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4382.559326] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4385.105903] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4387.878966] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 4396.000106] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4396.003059] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4396.008563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4398.053144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4398.054574] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4398.055095] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4401.070894] LustreError: 285281:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4401.147264] Lustre: server umount lustre-OST0000 complete [ 4403.168761] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4403.169428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4403.172388] Lustre: Skipped 1 previous similar message [ 4403.174807] Lustre: Skipped 1 previous similar message [ 4408.288926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4408.291184] Lustre: Skipped 1 previous similar message [ 4408.703486] LustreError: 285483:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4408.706044] LustreError: 285483:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4408.733425] Lustre: server umount lustre-MDT0000 complete [ 4410.125484] LustreError: 282854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754409719 with bad export cookie 17178505836954321062 [ 4410.127978] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4410.129159] LustreError: 282854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4410.197701] LustreError: 285685:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4410.200349] LustreError: 285685:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4410.295706] Lustre: server umount lustre-MDT0001 complete [ 4414.654946] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 4416.084190] Key type lgssc unregistered [ 4416.263526] LNet: 286211:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4416.266767] LNetError: 286211:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4416.273329] LNet: Removed LNI 192.168.203.133@tcp [ 4416.666284] Key type .llcrypt unregistered [ 4416.667862] Key type ._llcrypt unregistered [ 4421.571399] Key type ._llcrypt registered [ 4421.573439] Key type .llcrypt registered [ 4421.635245] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4428.634026] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 4434.985911] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 4435.277143] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4435.284070] alg: No test for adler32 (adler32-zlib) [ 4436.127715] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 4436.203270] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 4437.791109] Key type lgssc registered [ 4438.153768] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4441.131162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4443.550687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4445.748798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4447.831141] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4450.236570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4450.254894] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4450.260494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4451.340573] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4451.351966] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4451.389938] Lustre: lustre-MDT0000: new disk, initializing [ 4451.417980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4451.424720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4452.749978] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4457.090594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4457.117759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4457.167552] Lustre: 289675:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4457.191574] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4457.193215] Lustre: Skipped 1 previous similar message [ 4457.228248] Lustre: lustre-MDT0001: new disk, initializing [ 4457.262228] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4457.274089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4457.278157] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4458.651342] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4461.958987] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4463.237969] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4465.612754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4465.637967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4465.732900] Lustre: lustre-OST0000: new disk, initializing [ 4465.734692] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4465.764325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4467.793499] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4468.862460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4468.866501] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4468.902534] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4471.185740] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4472.686143] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4472.783779] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4474.322460] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4474.401616] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4477.102076] LustreError: 292582:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4477.169549] Lustre: server umount lustre-OST0000 complete [ 4479.456665] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4479.457579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4479.459638] Lustre: Skipped 1 previous similar message [ 4479.462858] Lustre: Skipped 1 previous similar message [ 4484.576273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4484.578586] Lustre: Skipped 1 previous similar message [ 4488.675785] LustreError: 292783:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4488.678386] LustreError: 292783:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4488.709166] Lustre: server umount lustre-MDT0000 complete [ 4490.015283] LustreError: 290450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754409799 with bad export cookie 7681751582706070533 [ 4490.018160] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4490.020497] LustreError: 290450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4499.426459] LustreError: 292984:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4499.428791] LustreError: 292984:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4499.501751] Lustre: server umount lustre-MDT0001 complete [ 4502.656777] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 12:03:31 (1754409811) [ 4505.213260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4505.356756] LustreError: 293655:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4505.364194] LustreError: 293655:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 4505.382883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4506.897569] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4510.068196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4510.185107] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4511.568313] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4513.605395] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4514.949540] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4517.346244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4519.673031] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4521.686384] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4540.392172] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4540.392310] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4540.395331] LustreError: Skipped 1 previous similar message [ 4540.400422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4540.403854] Lustre: Skipped 1 previous similar message [ 4541.550148] LustreError: 295906:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4541.552791] LustreError: 295906:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4541.613687] Lustre: server umount lustre-OST0000 complete [ 4545.505481] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4545.508883] Lustre: Skipped 1 previous similar message [ 4545.510961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4545.512410] Lustre: Skipped 1 previous similar message [ 4550.624571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4550.627540] Lustre: Skipped 1 previous similar message [ 4557.279307] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4557.319515] LustreError: 296108:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4557.322230] LustreError: 296108:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4557.356111] Lustre: server umount lustre-MDT0000 complete [ 4558.717142] LustreError: 293637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754409868 with bad export cookie 7681751582706071401 [ 4558.719591] LustreError: 294371:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 4558.720582] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4558.722283] LustreError: 293637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4558.879087] Lustre: server umount lustre-MDT0001 complete [ 4562.829847] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 4563.968061] Key type lgssc unregistered [ 4564.127421] LNet: 296836:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4564.130792] LNetError: 296836:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4564.138339] LNet: Removed LNI 192.168.203.133@tcp [ 4564.459832] Key type .llcrypt unregistered [ 4564.461612] Key type ._llcrypt unregistered [ 4573.319627] Key type ._llcrypt registered [ 4573.320792] Key type .llcrypt registered [ 4573.359401] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 4579.463217] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 4579.791222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4579.822106] alg: No test for adler32 (adler32-zlib) [ 4580.673697] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 4580.748285] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 4582.327169] Key type lgssc registered [ 4582.694706] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4585.598558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4587.794581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4589.881462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4592.010935] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4593.709406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4595.474979] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4597.791909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4597.809571] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4597.814667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4598.897210] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4598.910214] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4598.945786] Lustre: lustre-MDT0000: new disk, initializing [ 4598.973358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4598.980480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4600.262935] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4604.476850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4604.495787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4604.529094] Lustre: 300601:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4604.553650] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4604.556040] Lustre: Skipped 1 previous similar message [ 4604.599665] Lustre: lustre-MDT0001: new disk, initializing [ 4604.634384] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4604.648227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4604.652113] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4605.957967] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4609.047888] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4610.280220] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4612.587149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4612.609688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4612.696627] Lustre: lustre-OST2710: new disk, initializing [ 4612.698372] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4612.728107] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4614.653797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4614.658094] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4614.697514] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4614.738794] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4618.010130] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 4620.262158] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4620.286506] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4620.335105] Lustre: lustre-OST03e8: new disk, initializing [ 4620.336807] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4620.369849] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4620.828230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4620.831780] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4620.869675] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4622.279050] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4624.038423] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 4626.594717] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4626.668781] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4628.193818] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4628.273718] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4629.678450] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4629.762114] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4631.159394] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4631.251279] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4635.615966] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4635.619264] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4635.624545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4636.132854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4636.133704] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4636.135498] Lustre: Skipped 1 previous similar message [ 4636.138966] Lustre: Skipped 2 previous similar messages [ 4638.529589] LustreError: 304893:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4638.562693] Lustre: server umount lustre-MDT0000 complete [ 4639.882649] LustreError: 300636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754409949 with bad export cookie 16930794874647681844 [ 4639.887625] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4639.954964] LustreError: 305094:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4639.957282] LustreError: 305094:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4640.057489] Lustre: server umount lustre-MDT0001 complete [ 4651.696151] LustreError: 305294:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4651.697695] LustreError: 305294:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4651.715280] Lustre: server umount lustre-OST2710 complete [ 4663.281879] LustreError: 305496:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4663.283650] LustreError: 305496:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4663.370689] Lustre: server umount lustre-OST03e8 complete [ 4669.477806] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 4672.835704] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 4676.649560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4679.090099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4681.265435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4683.579555] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4685.956242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4685.980968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4686.085057] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4686.097230] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4686.133448] Lustre: lustre-MDT0000: new disk, initializing [ 4686.165264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4686.171746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4687.615091] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4692.077287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4692.104767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4692.140053] Lustre: 308276:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4692.257183] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4693.670345] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4697.048641] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4698.385808] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4700.783893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4700.806085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4700.898885] Lustre: lustre-OST0000: new disk, initializing [ 4700.900381] Lustre: Skipped 1 previous similar message [ 4700.902038] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4700.903592] Lustre: Skipped 2 previous similar messages [ 4700.933335] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4700.936481] Lustre: Skipped 1 previous similar message [ 4702.524852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4702.527110] Lustre: Skipped 1 previous similar message [ 4702.529453] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4702.564934] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4702.844917] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4706.076776] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4707.533992] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4707.619257] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4709.069993] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4709.147502] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4711.852601] LustreError: 311187:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4711.854646] LustreError: 311187:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4711.915354] Lustre: server umount lustre-OST0000 complete [ 4718.048795] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4718.049048] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4718.052804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4718.056420] Lustre: Skipped 2 previous similar messages [ 4723.168095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4723.170060] Lustre: Skipped 1 previous similar message [ 4723.171335] LustreError: 311387:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4723.173442] LustreError: 311387:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4723.200543] Lustre: server umount lustre-MDT0000 complete [ 4724.549143] LustreError: 308267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754410034 with bad export cookie 16930794874647683510 [ 4724.551942] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4724.554088] LustreError: 308267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4735.022044] Lustre: server umount lustre-MDT0001 complete [ 4737.944647] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 12:07:26 (1754410046) [ 4738.506603] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4739.108389] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 12:07:27 (1754410047) [ 4740.177305] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 4742.835595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4744.183312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4745.477719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4746.789820] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4747.173159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4749.304547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4749.378445] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4749.386455] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4749.443803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4750.708917] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4753.571298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4753.606833] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4753.615900] Lustre: 313708:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4755.126053] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4757.184793] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4758.518901] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4760.863344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4760.888319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4760.962188] Lustre: Denying initial registration attempt from nid 192.168.203.133@tcp, specified as failover [ 4760.964571] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4760.969173] LustreError: 315313:0:(tgt_mount.c:2419:server_fill_super()) Unable to start targets: -99 [ 4760.971252] LustreError: 315313:0:(tgt_mount.c:1913:server_put_super()) no obd lustre-OST0000 [ 4760.973152] LustreError: 315313:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 4761.023891] Lustre: server umount lustre-OST0000 complete [ 4761.025629] LustreError: 315313:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4764.127851] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4764.130366] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4764.134579] Lustre: Skipped 1 previous similar message [ 4764.137164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4769.404521] LustreError: 315680:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 4769.406114] LustreError: 315680:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 4769.427258] Lustre: server umount lustre-MDT0000 complete [ 4770.686428] LustreError: 313702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754410080 with bad export cookie 16930794874647684378 [ 4770.691979] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4770.825456] Lustre: server umount lustre-MDT0001 complete [ 4774.664834] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 4775.871507] Key type lgssc unregistered [ 4776.010411] LNet: 316408:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4776.013581] LNetError: 316408:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4776.023388] LNet: Removed LNI 192.168.203.133@tcp [ 4776.348049] Key type .llcrypt unregistered [ 4776.348982] Key type ._llcrypt unregistered [ 4780.290993] Key type ._llcrypt registered [ 4780.292548] Key type .llcrypt registered [ 4780.335525] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 4780.846878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4780.857731] alg: No test for adler32 (adler32-zlib) [ 4781.734679] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 4781.817675] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 4783.407141] Key type lgssc registered [ 4783.843946] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4786.472449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4787.926713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4789.284270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4790.651585] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4791.013245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4797.738698] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 4800.892424] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4800.899044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4802.005179] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4802.015920] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4802.124381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4803.583867] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4806.688599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4806.725505] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4806.739037] Lustre: 319127:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4806.846275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4808.442921] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4810.426573] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4811.665515] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4813.984590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4814.008865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4814.076880] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4814.147583] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4816.060782] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4817.785992] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4821.984487] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4821.985046] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4821.988739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4821.991388] Lustre: Skipped 1 previous similar message [ 4821.995931] Lustre: Skipped 1 previous similar message [ 4824.556434] LustreError: 321392:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4824.614681] Lustre: server umount lustre-OST0000 complete [ 4827.104415] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4827.105258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4827.108154] Lustre: Skipped 1 previous similar message [ 4831.997036] LustreError: 321594:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4831.999464] LustreError: 321594:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4832.029456] Lustre: server umount lustre-MDT0000 complete [ 4833.422456] LustreError: 319119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754410143 with bad export cookie 17617394124522273942 [ 4833.427518] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4833.430838] LustreError: 319119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4833.495438] LustreError: 321797:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4833.497076] LustreError: 321797:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4833.590898] Lustre: server umount lustre-MDT0001 complete [ 4837.396865] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 4838.512168] Key type lgssc unregistered [ 4838.645397] LNet: 322323:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4838.648142] LNetError: 322323:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4838.660252] LNet: Removed LNI 192.168.203.133@tcp [ 4838.979062] Key type .llcrypt unregistered [ 4838.979932] Key type ._llcrypt unregistered [ 4848.586815] Key type ._llcrypt registered [ 4848.588231] Key type .llcrypt registered [ 4848.630182] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 4848.989544] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4849.001905] alg: No test for adler32 (adler32-zlib) [ 4849.867203] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 4849.960701] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 4851.543198] Key type lgssc registered [ 4851.962507] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4854.731912] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4854.739550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4855.896102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4857.193706] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4860.082457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4860.202550] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4861.649817] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4863.629414] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4865.123890] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4867.648025] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4867.765364] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4869.698509] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4871.641734] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4885.983972] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4885.987535] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4885.992736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4887.300479] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.33@tcp (stopping) [ 4887.521676] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4890.811891] LustreError: 325936:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4890.834079] Lustre: server umount lustre-MDT0000 complete [ 4892.072227] LustreError: 323670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754410201 with bad export cookie 14045691264817831004 [ 4892.075461] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4892.076954] LustreError: 323670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4892.412152] LustreError: 325823:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4892.420080] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.33@tcp (stopping) [ 4892.420389] LustreError: 325823:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 4892.422772] Lustre: Skipped 3 previous similar messages [ 4892.640485] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4892.644965] Lustre: Skipped 1 previous similar message [ 4897.532508] LustreError: 325823:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4897.532955] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.33@tcp (stopping) [ 4897.539748] LustreError: 325823:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 4897.542741] Lustre: Skipped 1 previous similar message [ 4898.174603] LustreError: 326141:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4898.176744] LustreError: 326141:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4898.259470] Lustre: server umount lustre-MDT0001 complete [ 4900.751839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4903.948299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4917.215784] LustreError: 323000:0:(client.c:1385:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f98802a7 x1839632386908288/t0(0) o250->MGC192.168.203.133@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4917.338899] LustreError: 325284:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4917.348900] LustreError: 325284:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 4917.370542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4917.398247] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 4918.945401] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4921.440263] LustreError: 326953:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4921.449154] LustreError: 326953:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 4922.094602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4922.220678] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4923.608236] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 4925.564645] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4927.332444] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 4929.294946] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 4930.637956] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4931.334786] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4933.087957] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4933.090265] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4933.094839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4933.096767] Lustre: Skipped 1 previous similar message [ 4937.697640] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4937.698539] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4938.866115] LustreError: 328435:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4938.868648] LustreError: 328435:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4938.938085] Lustre: server umount lustre-OST0000 complete [ 4942.817605] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4942.819232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4942.823069] Lustre: Skipped 3 previous similar messages [ 4946.432368] LustreError: 328636:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4946.434450] LustreError: 328636:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4946.464364] Lustre: server umount lustre-MDT0000 complete [ 4947.857201] LustreError: 323671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754410257 with bad export cookie 14045691264817838158 [ 4947.860254] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4947.862056] LustreError: 323671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4948.017796] Lustre: server umount lustre-MDT0001 complete [ 4952.336719] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 4953.955397] Key type lgssc unregistered [ 4954.210541] LNet: 329363:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4954.229530] LNetError: 329363:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4954.245297] LNet: Removed LNI 192.168.203.133@tcp [ 4954.872849] Key type .llcrypt unregistered [ 4954.875095] Key type ._llcrypt unregistered [ 4981.805443] Key type ._llcrypt registered [ 4981.821967] Key type .llcrypt registered [ 4982.018414] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 4984.678425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4984.739070] alg: No test for adler32 (adler32-zlib) [ 4986.140657] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 4986.602587] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 4988.503147] Key type lgssc registered [ 4990.229785] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5001.488286] hrtimer: interrupt took 4278041 ns [ 5002.923116] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5002.985052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5005.015738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5012.614380] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5014.497161] LustreError: 330756:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5014.515192] LustreError: 330756:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 5019.633480] LustreError: 330756:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5024.744952] LustreError: 330756:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5026.459887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5027.210411] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5032.838277] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5041.494182] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5045.805745] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5049.356084] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5049.366270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5049.390049] Lustre: Skipped 1 previous similar message [ 5051.424943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5051.434664] Lustre: Skipped 1 previous similar message [ 5055.636349] LustreError: 332138:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5055.777700] Lustre: server umount lustre-MDT0000 complete [ 5056.483981] LustreError: 330756:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5056.497719] LustreError: 330756:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 5059.627263] LustreError: 332139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754410369 with bad export cookie 8295877366354411737 [ 5059.637466] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5059.646171] LustreError: 332139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5059.764137] LustreError: 332341:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5059.767463] LustreError: 332341:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5060.054971] Lustre: server umount lustre-MDT0001 complete [ 5074.161901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5074.572601] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5074.633283] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5074.905094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5078.903894] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5086.359163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5086.438375] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5086.460692] Lustre: 333110:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5086.676437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5089.609831] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5094.422137] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5097.354712] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5103.710770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5103.874316] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 5103.887509] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 5103.900366] LustreError: 334757:0:(tgt_mount.c:2419:server_fill_super()) Unable to start targets: -2 [ 5103.904180] LustreError: 334757:0:(tgt_mount.c:1913:server_put_super()) no obd lustre-OST0000 [ 5103.914651] LustreError: 334757:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 5104.039486] Lustre: server umount lustre-OST0000 complete [ 5104.049568] LustreError: 334757:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5111.107551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5111.294811] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5111.472625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5116.706650] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5118.098330] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 5121.740649] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5129.253557] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5129.329446] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5129.412994] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5129.473089] Lustre: lustre-OST0001: new disk, initializing [ 5129.479668] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5129.577972] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5135.009232] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5136.032037] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5136.057465] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5136.182620] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5140.779851] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5146.601191] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5146.603604] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5146.608328] Lustre: Skipped 1 previous similar message [ 5146.628609] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5149.703337] LustreError: 336865:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5149.706173] LustreError: 336865:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5149.820749] Lustre: server umount lustre-OST0001 complete [ 5151.712392] LustreError: 335224:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5151.727463] LustreError: 335224:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 5153.762050] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5153.770674] LustreError: Skipped 1 previous similar message [ 5153.772644] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5153.786679] Lustre: Skipped 1 previous similar message [ 5153.793674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5153.796596] Lustre: Skipped 1 previous similar message [ 5161.956211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5161.956972] LustreError: 335224:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5161.976911] Lustre: Skipped 4 previous similar messages [ 5161.993446] LustreError: 335224:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 5168.095549] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5168.125466] LustreError: 337068:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5168.130049] LustreError: 337068:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5168.307166] Lustre: server umount lustre-OST0000 complete [ 5172.194959] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5172.202610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5172.207322] Lustre: Skipped 2 previous similar messages [ 5172.217204] Lustre: Skipped 3 previous similar messages [ 5177.839893] LustreError: 337271:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5177.846844] LustreError: 337271:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5177.961037] Lustre: server umount lustre-MDT0000 complete [ 5181.130167] LustreError: 336178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754410490 with bad export cookie 8295877366354412283 [ 5181.131131] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5181.148435] LustreError: 336178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5181.683890] Lustre: server umount lustre-MDT0001 complete [ 5190.021630] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 5192.396264] Key type lgssc unregistered [ 5192.633593] LNet: 338000:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5192.640344] LNetError: 338000:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5192.656639] LNet: Removed LNI 192.168.203.133@tcp [ 5193.534334] Key type .llcrypt unregistered [ 5193.536047] Key type ._llcrypt unregistered [ 5223.174320] Key type ._llcrypt registered [ 5223.175672] Key type .llcrypt registered [ 5223.281082] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 5236.082882] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 5237.000829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5237.034125] alg: No test for adler32 (adler32-zlib) [ 5238.143324] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 5238.356718] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 5240.047195] Key type lgssc registered [ 5240.992604] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5248.733151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5253.936358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5259.805925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5266.558471] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5273.167141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5273.212769] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5273.238974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5274.481985] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5274.541679] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5274.618715] Lustre: lustre-MDT0000: new disk, initializing [ 5274.691539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5274.706750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5278.277203] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5286.801907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5286.851872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5286.917313] Lustre: 341938:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5286.967161] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5286.971485] Lustre: Skipped 1 previous similar message [ 5287.028853] Lustre: lustre-MDT0001: new disk, initializing [ 5287.105432] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5287.143855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5287.155614] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5290.289789] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5296.978241] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5300.207973] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5306.132898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5306.201916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5306.389805] Lustre: lustre-OST0000: new disk, initializing [ 5306.404547] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5306.480221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5310.951863] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5314.132286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5314.139666] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5314.233634] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5318.489101] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5322.309116] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5322.562891] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5326.437602] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5326.645137] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5329.890672] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5329.895739] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5329.912833] LustreError: Skipped 1 previous similar message [ 5329.914482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5329.937269] Lustre: Skipped 1 previous similar message [ 5334.398409] LustreError: 344844:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5334.581144] Lustre: server umount lustre-OST0000 complete [ 5338.594459] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5338.605532] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5338.630419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5338.634946] Lustre: Skipped 1 previous similar message [ 5340.131138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5340.134157] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5344.102463] LustreError: 345045:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5344.113639] LustreError: 345045:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5346.147624] LustreError: 345045:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 5346.152116] LustreError: 345045:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5346.195735] Lustre: server umount lustre-MDT0000 complete [ 5349.192018] LustreError: 341930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754410658 with bad export cookie 2220892158635052675 [ 5349.194364] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5349.209846] LustreError: 341930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5349.365760] LustreError: 345248:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5349.370725] LustreError: 345248:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5349.569340] Lustre: server umount lustre-MDT0001 complete [ 5356.492957] Lustre: DEBUG MARKER: == conf-sanity test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 12:17:44 (1754410664) [ 5361.188086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5366.406130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5382.695866] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 5388.635868] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 5395.458140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5399.436842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5402.807719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5406.665780] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5410.317665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5410.353579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5410.473229] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5410.495374] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5410.542502] Lustre: lustre-MDT0000: new disk, initializing [ 5410.599893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5410.614811] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5412.551981] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5417.974261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5418.008493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5418.054788] Lustre: 349009:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5418.086318] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5418.089128] Lustre: Skipped 1 previous similar message [ 5418.129565] Lustre: lustre-MDT0001: new disk, initializing [ 5418.174322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5418.194970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5418.200329] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5420.180890] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5424.778751] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5426.922586] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5430.796392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5430.829916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5430.987704] Lustre: lustre-OST0000: new disk, initializing [ 5430.990768] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5431.035733] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5433.443103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5433.453489] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5433.506688] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5434.130395] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5438.200545] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5439.910702] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5440.010607] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5441.564263] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5441.661905] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5444.065881] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5444.067041] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5444.069054] LustreError: Skipped 1 previous similar message [ 5444.075405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5444.079201] Lustre: Skipped 2 previous similar messages [ 5448.491120] LustreError: 351916:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5448.494313] LustreError: 351916:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5448.563815] Lustre: server umount lustre-OST0000 complete [ 5454.305382] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5454.305526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5454.306543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5454.316515] Lustre: Skipped 2 previous similar messages [ 5456.102343] Lustre: server umount lustre-MDT0000 complete [ 5457.693887] LustreError: 350744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754410767 with bad export cookie 2220892158635053557 [ 5457.697288] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5457.700746] LustreError: 350744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5457.782409] LustreError: 352319:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5457.786794] LustreError: 352319:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 5457.887128] Lustre: server umount lustre-MDT0001 complete [ 5461.204528] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 12:19:30 (1754410770) [ 5464.011711] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 12:19:32 (1754410772) [ 5467.258574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5467.439031] LustreError: 353331:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5467.446936] LustreError: 353331:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 5467.468936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5468.915697] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5472.220299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5473.912837] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5476.017302] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5477.422523] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5480.200071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5482.596762] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5484.607589] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5498.822795] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 5499.598127] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5500.399588] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5501.052134] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 5501.854589] Lustre: Failing over lustre-MDT0000 [ 5501.975646] LustreError: 355771:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5501.978509] LustreError: 355771:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5502.018303] Lustre: server umount lustre-MDT0000 complete [ 5502.945831] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5502.955391] LustreError: 353349:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5502.961435] LustreError: 353349:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 5505.286655] LustreError: 353327:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5508.064682] LustreError: 353349:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5508.071085] LustreError: 353349:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 5513.184803] LustreError: 353349:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5513.193618] LustreError: 353349:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 5515.138650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5515.180312] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5515.274000] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5515.276654] Lustre: Skipped 2 previous similar messages [ 5515.297382] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5515.516201] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5516.644144] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5520.355898] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 5520.360884] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 5520.377916] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5521.651090] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5522.254325] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5524.148461] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 5524.826163] Lustre: Failing over lustre-MDT0000 [ 5525.028949] Lustre: server umount lustre-MDT0000 complete [ 5525.473121] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5525.474074] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5525.474651] LustreError: 353325:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5525.477478] Lustre: Skipped 4 previous similar messages [ 5538.012759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5538.057976] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5538.178129] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5539.521166] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5541.116739] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5543.396420] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 5543.400200] Lustre: Skipped 2 previous similar messages [ 5543.407202] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 5543.422721] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5544.670506] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5545.217849] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5548.514233] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5548.514694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5548.519756] Lustre: Skipped 3 previous similar messages [ 5553.596681] LustreError: 357871:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5553.599123] LustreError: 357871:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 5553.626530] Lustre: server umount lustre-MDT0000 complete [ 5553.633675] LustreError: 353330:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5553.639412] LustreError: 353330:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 13 previous similar messages [ 5554.889636] LustreError: 353312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754410864 with bad export cookie 2220892158635056238 [ 5554.894000] LustreError: 353312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5554.894075] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5555.045905] Lustre: server umount lustre-MDT0001 complete [ 5566.704524] Lustre: server umount lustre-OST0000 complete [ 5571.805026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5573.261079] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5576.105626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5577.574586] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5579.394921] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5580.645745] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5582.910604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5583.017727] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5583.019618] Lustre: Skipped 3 previous similar messages [ 5584.936293] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5586.721471] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5597.349558] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5601.760937] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5601.764209] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5601.767885] Lustre: Skipped 3 previous similar messages [ 5601.769933] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5604.836377] Lustre: server umount lustre-OST0000 complete [ 5612.244385] Lustre: server umount lustre-MDT0000 complete [ 5613.023884] LustreError: 359117:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5613.027750] LustreError: 359117:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 5613.490463] LustreError: 359079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754410923 with bad export cookie 2220892158635056945 [ 5613.494099] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5613.494422] LustreError: 359079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5613.629250] Lustre: server umount lustre-MDT0001 complete [ 5617.137956] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 5618.174947] Key type lgssc unregistered [ 5618.314367] LNet: 362277:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5618.316696] LNetError: 362277:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5618.322369] LNet: Removed LNI 192.168.203.133@tcp [ 5618.625021] Key type .llcrypt unregistered [ 5618.625868] Key type ._llcrypt unregistered [ 5628.550835] Key type ._llcrypt registered [ 5628.551789] Key type .llcrypt registered [ 5628.588381] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 5628.875386] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5628.882709] alg: No test for adler32 (adler32-zlib) [ 5629.735399] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 5629.823919] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 5631.407147] Key type lgssc registered [ 5631.746060] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5634.238715] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5634.245267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5635.386471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5636.649747] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5639.486898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5639.593988] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5640.717260] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5642.374249] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5643.469207] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5645.645562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5645.748374] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5647.662927] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5649.620393] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5660.073866] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5665.249783] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5665.250356] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5665.251388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5665.254159] Lustre: Skipped 2 previous similar messages [ 5670.368305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5670.370606] Lustre: Skipped 3 previous similar messages [ 5670.779921] LustreError: 365990:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5670.813698] Lustre: server umount lustre-MDT0000 complete [ 5672.122396] LustreError: 365891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754410981 with bad export cookie 7895917424402803317 [ 5672.126695] LustreError: 365891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5672.127580] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5672.191926] LustreError: 366193:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5672.194427] LustreError: 366193:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5672.289353] Lustre: server umount lustre-MDT0001 complete [ 5683.826140] LustreError: 366393:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5683.828658] LustreError: 366393:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5683.890716] Lustre: server umount lustre-OST0000 complete [ 5688.287654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5688.428929] LustreError: 367123:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5688.433871] LustreError: 367123:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 5688.449089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5689.830292] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5692.804242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5692.917699] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5694.214535] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5695.926218] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5697.138116] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5699.496244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5701.577964] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5703.498494] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5713.959680] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5723.615803] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5723.618700] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5723.622848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5727.101424] LustreError: 369473:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5727.103694] LustreError: 369473:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5727.133775] Lustre: server umount lustre-MDT0000 complete [ 5728.420260] LustreError: 367102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754411038 with bad export cookie 7895917424402804416 [ 5728.423974] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5728.424196] LustreError: 367102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5728.563280] Lustre: server umount lustre-MDT0001 complete [ 5740.143677] LustreError: 369875:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5740.145451] LustreError: 369875:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 5740.230134] Lustre: server umount lustre-OST0000 complete [ 5745.612505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5745.757076] LustreError: 370702:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5745.781300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5745.783566] Lustre: Skipped 1 previous similar message [ 5747.109163] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5750.139665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5751.665081] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5753.540883] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5754.799063] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5757.249555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5759.594896] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5761.652430] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5771.749903] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5774.815843] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5774.819241] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5774.824735] Lustre: Skipped 2 previous similar messages [ 5774.826880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5774.829332] Lustre: Skipped 3 previous similar messages [ 5776.865417] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5776.866078] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5779.436118] LustreError: 372951:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5779.438482] LustreError: 372951:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5779.501314] Lustre: server umount lustre-OST0000 complete [ 5780.959726] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5780.962262] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5780.966482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5780.967752] Lustre: Skipped 2 previous similar messages [ 5792.224662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5792.227286] Lustre: Skipped 5 previous similar messages [ 5795.295104] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5795.363562] Lustre: server umount lustre-MDT0000 complete [ 5796.735613] LustreError: 370682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754411106 with bad export cookie 7895917424402805424 [ 5796.739201] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5796.739878] LustreError: 370682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5796.806573] LustreError: 373356:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5796.809416] LustreError: 373356:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 5796.892814] Lustre: server umount lustre-MDT0001 complete [ 5800.728345] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 5801.865271] Key type lgssc unregistered [ 5801.995359] LNet: 373882:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5801.997217] LNetError: 373882:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5802.007292] LNet: Removed LNI 192.168.203.133@tcp [ 5802.342471] Key type .llcrypt unregistered [ 5802.343339] Key type ._llcrypt unregistered [ 5811.898045] Key type ._llcrypt registered [ 5811.899323] Key type .llcrypt registered [ 5811.947779] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 5812.277428] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5812.304090] alg: No test for adler32 (adler32-zlib) [ 5813.168544] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 5813.261556] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 5814.839161] Key type lgssc registered [ 5815.225025] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5817.940858] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5817.947077] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5817.950242] LustreError: 375271:0:(osd_handler.c:8374:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5817.959541] LustreError: 375271:0:(obd_config.c:779:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5817.962183] LustreError: 375271:0:(obd_mount.c:193:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5817.965251] LustreError: 375271:0:(tgt_mount.c:2387:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5817.969339] LustreError: 375271:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5820.582897] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5820.584772] LustreError: 375670:0:(osd_handler.c:8374:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5820.592050] LustreError: 375670:0:(obd_config.c:779:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5820.594457] LustreError: 375670:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5820.597673] LustreError: 375670:0:(tgt_mount.c:2387:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5820.599697] LustreError: 375670:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5827.584380] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 5828.788917] Key type lgssc unregistered [ 5828.926635] LNet: 376648:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5828.929687] LNetError: 376648:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5828.940393] LNet: Removed LNI 192.168.203.133@tcp [ 5829.245941] Key type .llcrypt unregistered [ 5829.247031] Key type ._llcrypt unregistered [ 5835.326461] Key type ._llcrypt registered [ 5835.327738] Key type .llcrypt registered [ 5835.365355] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 5842.152915] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 5842.592466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5842.629339] alg: No test for adler32 (adler32-zlib) [ 5843.509969] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 5843.607213] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 5845.207195] Key type lgssc registered [ 5845.640235] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5849.236775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5852.003230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5854.317862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5856.797370] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5859.434182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5859.454950] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5859.461739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5860.555546] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5860.569914] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5860.613264] Lustre: lustre-MDT0000: new disk, initializing [ 5860.645481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5860.653824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5862.116142] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5866.636560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5866.666059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5866.708535] Lustre: 379741:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5866.740508] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5866.743447] Lustre: Skipped 1 previous similar message [ 5866.782320] Lustre: lustre-MDT0001: new disk, initializing [ 5866.825731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5866.842895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5866.848416] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5868.321664] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5871.712733] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5873.002866] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5875.375153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5875.401747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5875.494751] Lustre: lustre-OST0000: new disk, initializing [ 5875.497220] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5875.527146] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5877.610107] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5879.098486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5879.102583] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5879.139911] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5880.871476] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5882.287980] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5882.364828] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5883.750341] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5883.837409] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5886.572078] LustreError: 382648:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5886.636932] Lustre: server umount lustre-OST0000 complete [ 5889.505123] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5889.505945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5889.510149] Lustre: Skipped 1 previous similar message [ 5889.514839] Lustre: Skipped 1 previous similar message [ 5894.624142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5894.626439] Lustre: Skipped 1 previous similar message [ 5899.744944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5899.747390] Lustre: Skipped 1 previous similar message [ 5902.303129] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5902.337022] LustreError: 382849:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5902.339096] LustreError: 382849:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5902.368283] Lustre: server umount lustre-MDT0000 complete [ 5903.735208] LustreError: 381475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754411213 with bad export cookie 8117728481474313780 [ 5903.738740] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5903.739371] LustreError: 381475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5914.084468] LustreError: 383053:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5914.087581] LustreError: 383053:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5914.181155] Lustre: server umount lustre-MDT0001 complete [ 5917.413924] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5918.057704] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 12:27:06 (1754411226) [ 5920.556864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5920.766715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5922.173162] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5925.316455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5925.464715] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5926.983523] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5929.088704] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5930.419971] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5932.787114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5934.927506] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5936.812383] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5939.212247] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5939.234350] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5939.289523] Lustre: lustre-OST0001: new disk, initializing [ 5939.291361] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5939.321424] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5939.324022] Lustre: Skipped 1 previous similar message [ 5941.363180] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 5942.656782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5942.659636] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5942.683629] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5944.628088] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5947.492036] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5947.494281] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5947.496930] Lustre: Skipped 1 previous similar message [ 5952.558077] LustreError: 386948:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5952.560613] LustreError: 386948:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5952.579760] Lustre: server umount lustre-OST0001 complete [ 5952.998308] LustreError: 386701:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5953.003819] LustreError: 386701:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 5958.114964] LustreError: 385357:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5958.120515] LustreError: 385357:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 5963.235708] LustreError: 385357:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5963.240414] LustreError: 385357:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 5966.816028] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5966.818407] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5966.823829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5966.826039] Lustre: Skipped 2 previous similar messages [ 5968.354324] LustreError: 385357:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5968.355104] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5968.359301] LustreError: 385357:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 5970.733095] LustreError: 387172:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5970.735207] LustreError: 387172:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5970.796361] Lustre: server umount lustre-OST0000 complete [ 5973.472780] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5973.475771] Lustre: Skipped 2 previous similar messages [ 5978.261913] Lustre: server umount lustre-MDT0000 complete [ 5978.593411] LustreError: 383773:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5978.598098] LustreError: 383773:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 5979.552319] LustreError: 383754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754411289 with bad export cookie 8117728481474314648 [ 5979.555176] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5979.556743] LustreError: 383754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5985.663436] LustreError: 387575:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 5985.665995] LustreError: 387575:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 5985.756462] Lustre: server umount lustre-MDT0001 complete [ 5989.882230] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 5991.063719] Key type lgssc unregistered [ 5991.207407] LNet: 388102:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5991.211158] LNetError: 388102:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5991.218445] LNet: Removed LNI 192.168.203.133@tcp [ 5991.599139] Key type .llcrypt unregistered [ 5991.600740] Key type ._llcrypt unregistered [ 6003.230271] Key type ._llcrypt registered [ 6003.231778] Key type .llcrypt registered [ 6003.275805] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 6009.838413] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 6010.222981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6010.240937] alg: No test for adler32 (adler32-zlib) [ 6011.090289] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 6011.174286] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 6012.751156] Key type lgssc registered [ 6013.122634] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6016.275213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6018.667755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6020.823281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6023.003922] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6025.324454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6025.345068] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6025.350680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6026.436115] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6026.448879] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6026.485387] Lustre: lustre-MDT0000: new disk, initializing [ 6026.511886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6026.520539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6027.868188] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6031.985269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6032.002834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6032.033473] Lustre: 392009:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6032.055214] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6032.056747] Lustre: Skipped 1 previous similar message [ 6032.088439] Lustre: lustre-MDT0001: new disk, initializing [ 6032.118360] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6032.129978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6032.133102] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6033.388055] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6036.485126] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6037.698533] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6039.950732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6039.972687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6040.065444] Lustre: lustre-OST0000: new disk, initializing [ 6040.067204] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6040.092754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6041.963318] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6042.740694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6042.745207] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6042.771404] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6044.912156] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6046.290467] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6046.361518] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6047.557065] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6047.633105] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6048.225294] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6048.225372] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6048.226380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6048.227824] Lustre: Skipped 1 previous similar message [ 6048.229504] LustreError: Skipped 1 previous similar message [ 6053.345218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6053.348065] Lustre: Skipped 2 previous similar messages [ 6054.315097] LustreError: 394913:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6054.380979] Lustre: server umount lustre-OST0000 complete [ 6057.951819] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6057.954091] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6057.957879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6061.688686] LustreError: 395118:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6061.690123] LustreError: 395118:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6061.714622] Lustre: server umount lustre-MDT0000 complete [ 6062.851953] LustreError: 392783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754411372 with bad export cookie 8451690470816044802 [ 6062.854400] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6062.854637] LustreError: 392783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6062.911641] LustreError: 395321:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6062.913070] LustreError: 395321:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6062.981202] Lustre: server umount lustre-MDT0001 complete [ 6065.532868] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 12:29:34 (1754411374) [ 6068.358287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6071.008709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6071.134250] Lustre: lustre-MDT0000: new disk, initializing [ 6071.163540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6072.469031] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6075.252768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6075.372097] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6076.686487] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6078.391618] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6079.583755] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6080.222469] LustreError: 396616:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6080.224557] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6080.225153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6080.225158] Lustre: Skipped 2 previous similar messages [ 6080.226278] LustreError: 396616:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6080.230070] Lustre: Skipped 2 previous similar messages [ 6080.236442] LustreError: 396616:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 6085.344156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6085.346195] Lustre: Skipped 2 previous similar messages [ 6086.329175] LustreError: 397916:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6086.331292] LustreError: 397916:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6086.354289] Lustre: server umount lustre-MDT0000 complete [ 6087.546208] LustreError: 396611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754411397 with bad export cookie 8451690470816045670 [ 6087.549173] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6087.551458] LustreError: 396611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6087.686611] Lustre: server umount lustre-MDT0001 complete [ 6090.640814] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 12:29:59 (1754411399) [ 6092.683384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6092.806470] LustreError: 398840:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6092.813237] LustreError: 398840:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 6092.827896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6094.159268] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6096.998182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6098.465926] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6100.314568] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6101.589301] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6103.919432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6106.050934] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6107.916249] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6126.521435] Lustre: Permanently deactivating lustre-OST0000 [ 6126.527097] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6132.721649] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6133.655475] LustreError: 401534:0:(mgs_llog.c:1937:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 6133.660248] LustreError: 401534:0:(mgs_llog.c:1991:mgs_replace_nids()) Only MGS is allowed to be started [ 6133.663395] LustreError: 401534:0:(mgs_handler.c:1161:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6134.310935] LustreError: 401583:0:(mgs_llog.c:1991:mgs_replace_nids()) Only MGS is allowed to be started [ 6134.313746] LustreError: 401583:0:(mgs_handler.c:1161:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6141.036426] LustreError: 401680:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6141.038946] LustreError: 401680:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6141.104934] Lustre: server umount lustre-OST0000 complete [ 6142.042903] LustreError: 401832:0:(mgs_llog.c:1991:mgs_replace_nids()) Only MGS is allowed to be started [ 6142.045038] LustreError: 401832:0:(mgs_handler.c:1161:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6142.944194] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6142.944852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6142.947864] Lustre: Skipped 1 previous similar message [ 6153.190640] LustreError: 401929:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6153.193725] LustreError: 401929:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6153.225502] Lustre: server umount lustre-MDT0000 complete [ 6154.436032] LustreError: 398820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754411464 with bad export cookie 8451690470816046202 [ 6154.438705] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6154.440931] LustreError: 398820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6165.043984] Lustre: server umount lustre-MDT0001 complete [ 6167.955948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6169.421208] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6170.357704] LustreError: 402968:0:(mgs_handler.c:1161:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 6172.571965] Lustre: 403307:0:(mgs_llog.c:1698:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.133@tcp [ 6173.295594] LustreError: 403404:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6173.297414] LustreError: 403404:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6173.320715] Lustre: server umount MGS complete [ 6177.285483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6177.441433] LustreError: 404032:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6177.447391] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6177.451055] Lustre: Skipped 1 previous similar message [ 6177.472773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6177.475511] Lustre: Skipped 2 previous similar messages [ 6178.903127] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6182.009310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6182.113137] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6183.490241] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6185.335279] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6186.530546] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6187.172036] Lustre: Permanently reactivating lustre-OST0000 [ 6187.176577] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6187.178787] Lustre: Skipped 2 previous similar messages [ 6215.135140] LustreError: 405673:0:(osp_dev.c:694:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 6218.096680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6218.215054] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6218.216857] Lustre: Skipped 1 previous similar message [ 6220.049697] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6221.792317] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6234.593140] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6234.594887] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6234.595288] LustreError: Skipped 1 previous similar message [ 6234.598786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6234.601248] Lustre: Skipped 6 previous similar messages [ 6235.691107] LustreError: 407494:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6235.694864] LustreError: 407494:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 6235.761454] Lustre: server umount lustre-OST0000 complete [ 6238.687848] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6243.159521] Lustre: server umount lustre-MDT0000 complete [ 6244.410091] LustreError: 404012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754411554 with bad export cookie 8451690470816047147 [ 6244.413099] LustreError: 404012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6244.413978] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6244.551963] Lustre: server umount lustre-MDT0001 complete [ 6248.187092] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 6249.326481] Key type lgssc unregistered [ 6249.463378] LNet: 408429:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6249.467055] LNetError: 408429:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6249.476263] LNet: Removed LNI 192.168.203.133@tcp [ 6249.785715] Key type .llcrypt unregistered [ 6249.786724] Key type ._llcrypt unregistered [ 6255.906964] Key type ._llcrypt registered [ 6255.908017] Key type .llcrypt registered [ 6255.950829] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 6262.468414] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 6262.912149] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6262.925135] alg: No test for adler32 (adler32-zlib) [ 6263.794325] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 6263.892129] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 6265.479166] Key type lgssc registered [ 6265.868323] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6268.952866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6271.240878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6273.390373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6275.669137] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6278.078817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6278.103042] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6278.108300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6279.191082] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6279.201580] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6279.239358] Lustre: lustre-MDT0000: new disk, initializing [ 6279.263736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6279.271264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6280.581265] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6284.781943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6284.806493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6284.842260] Lustre: 411519:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6284.869248] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6284.873244] Lustre: Skipped 1 previous similar message [ 6284.919787] Lustre: lustre-MDT0001: new disk, initializing [ 6284.957114] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6284.971415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6284.976119] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6286.380474] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6289.635763] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6290.967894] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6293.402513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6293.424971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6293.528520] Lustre: lustre-OST0000: new disk, initializing [ 6293.530220] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6293.561088] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6295.538976] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6298.788001] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6300.258585] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6301.182529] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6301.185744] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6301.223118] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6301.383113] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 6302.778431] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6302.869649] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6306.272840] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6306.276038] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6306.281503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6306.283679] Lustre: Skipped 1 previous similar message [ 6309.613306] LustreError: 414434:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6309.703236] Lustre: server umount lustre-OST0000 complete [ 6311.392617] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6311.393200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6311.396855] Lustre: Skipped 2 previous similar messages [ 6316.512815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6316.515133] Lustre: Skipped 2 previous similar messages [ 6317.180565] LustreError: 414635:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6317.184015] LustreError: 414635:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6317.215823] Lustre: server umount lustre-MDT0000 complete [ 6318.482279] LustreError: 412292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754411628 with bad export cookie 368511381535687526 [ 6318.485262] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6318.486023] LustreError: 412292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6318.555242] LustreError: 414838:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6318.557090] LustreError: 414838:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6318.644583] Lustre: server umount lustre-MDT0001 complete [ 6321.643790] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 6322.221324] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 12:33:51 (1754411631) [ 6324.548520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6324.739444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6326.204258] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6329.202894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6329.327093] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6330.743416] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6332.659224] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6333.907748] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6336.256540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6338.256968] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6340.196355] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6340.841494] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 6351.695539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 6353.887857] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6353.889941] LustreError: Skipped 1 previous similar message [ 6353.891117] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6353.894855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6354.914174] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6359.210719] LustreError: 417995:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6359.212405] LustreError: 417995:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6359.275329] Lustre: server umount lustre-OST0000 complete [ 6365.152638] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6365.152952] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6365.153617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6365.153623] Lustre: Skipped 2 previous similar messages [ 6365.167509] Lustre: Skipped 2 previous similar messages [ 6366.652304] LustreError: 418196:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6366.653990] LustreError: 418196:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6366.677764] Lustre: server umount lustre-MDT0000 complete [ 6367.929345] LustreError: 415538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754411677 with bad export cookie 368511381535688394 [ 6367.931872] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6367.935255] LustreError: 415538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6368.089427] Lustre: server umount lustre-MDT0001 complete [ 6371.830570] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 6373.065365] Key type lgssc unregistered [ 6373.227453] LNet: 418925:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6373.229537] LNetError: 418925:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6373.238462] LNet: Removed LNI 192.168.203.133@tcp [ 6373.562913] Key type .llcrypt unregistered [ 6373.563924] Key type ._llcrypt unregistered [ 6381.181208] Key type ._llcrypt registered [ 6381.182401] Key type .llcrypt registered [ 6381.224463] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 6381.354554] Key type .llcrypt unregistered [ 6381.355875] Key type ._llcrypt unregistered [ 6388.907229] Key type ._llcrypt registered [ 6388.908977] Key type .llcrypt registered [ 6388.948795] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 6389.308226] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6389.376704] alg: No test for adler32 (adler32-zlib) [ 6390.238463] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 6390.325248] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 6391.911205] Key type lgssc registered [ 6392.280228] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6394.952613] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6394.960624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6396.134247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6397.581764] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6400.594746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6400.705049] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6402.575621] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6404.405029] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6405.856874] LustreError: 421060:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6405.862634] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6405.864422] LustreError: 421060:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 6406.805840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6406.934180] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6408.348503] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6416.351914] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6416.355381] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6416.359509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6417.379184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6417.379775] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6417.382863] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6421.803679] LustreError: 423095:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 6421.866352] Lustre: server umount lustre-OST0000 complete [ 6427.106102] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 6427.109946] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6427.115778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6427.117716] Lustre: Skipped 1 previous similar message [ 6429.246139] LustreError: 423297:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6429.248106] LustreError: 423297:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6429.275252] Lustre: server umount lustre-MDT0000 complete [ 6430.545246] LustreError: 421041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754411740 with bad export cookie 8593197068389734025 [ 6430.548745] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6430.550765] LustreError: 421041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6430.613691] LustreError: 423500:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6430.615390] LustreError: 423500:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6430.690799] Lustre: server umount lustre-MDT0001 complete [ 6434.740711] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 6435.950176] Key type lgssc unregistered [ 6436.100407] LNet: 424026:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6436.102792] LNetError: 424026:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6436.114400] LNet: Removed LNI 192.168.203.133@tcp [ 6436.470694] Key type .llcrypt unregistered [ 6436.472554] Key type ._llcrypt unregistered [ 6445.975538] Key type ._llcrypt registered [ 6445.976984] Key type .llcrypt registered [ 6446.013954] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 6446.429577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6446.449689] alg: No test for adler32 (adler32-zlib) [ 6447.308961] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 6447.390489] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 6448.975173] Key type lgssc registered [ 6449.335874] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6452.077245] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6452.083156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6469.535242] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6475.679266] LustreError: 425381:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.133@tcp: failed processing log, type 4: rc = -110 [ 6505.375230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6507.309397] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6509.189569] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6511.570183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6511.748543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6511.776085] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6513.073213] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6515.986135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6516.116414] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6517.519740] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6519.575682] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6520.970380] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6524.386701] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6524.391729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6524.393618] Lustre: Skipped 1 previous similar message [ 6529.453091] LustreError: 427644:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 6529.515557] Lustre: server umount lustre-OST0000 complete [ 6531.551781] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6531.553963] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6531.558527] Lustre: Skipped 1 previous similar message [ 6531.560660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6534.624033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6534.624811] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6536.898274] LustreError: 427845:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 6536.900162] LustreError: 427845:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6536.929406] Lustre: server umount lustre-MDT0000 complete [ 6538.283470] LustreError: 425388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754411847 with bad export cookie 16571770484605715246 [ 6538.286482] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6538.290089] LustreError: 425388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6538.361962] LustreError: 428048:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6538.365635] LustreError: 428048:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6538.474023] Lustre: server umount lustre-MDT0001 complete [ 6542.295941] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 6543.393639] Key type lgssc unregistered [ 6543.538378] LNet: 428574:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6543.541460] LNetError: 428574:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6543.553315] LNet: Removed LNI 192.168.203.133@tcp [ 6543.871995] Key type .llcrypt unregistered [ 6543.873255] Key type ._llcrypt unregistered [ 6553.573816] Key type ._llcrypt registered [ 6553.574941] Key type .llcrypt registered [ 6553.622763] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 6554.055660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6554.095760] alg: No test for adler32 (adler32-zlib) [ 6554.978096] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 6555.077540] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 6556.671148] Key type lgssc registered [ 6557.046140] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6559.707530] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6559.713993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6560.870481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6562.190224] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6565.073063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6565.213149] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6566.649755] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6568.595222] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6569.854891] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6572.133706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6572.258847] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6574.169282] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6575.984925] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6586.598341] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6590.944523] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6590.947554] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6590.951428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6591.749557] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.33@tcp (stopping) [ 6591.971400] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6591.976076] Lustre: Skipped 1 previous similar message [ 6593.659775] LustreError: 432189:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6593.687570] Lustre: server umount lustre-MDT0000 complete [ 6594.517266] LustreError: 430655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.33@tcp arrived at 1754411904 with bad export cookie 3949819658957382424 [ 6594.522131] LustreError: 430655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6595.227079] LustreError: 430655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754411904 with bad export cookie 3949819658957381801 [ 6595.230856] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6595.231417] LustreError: 430655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6596.063858] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6596.066256] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6596.070415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6596.071826] Lustre: Skipped 3 previous similar messages [ 6597.087844] LustreError: 429942:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6597.092423] LustreError: 429942:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 6600.095914] LustreError: 431244:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6600.096311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6600.101735] LustreError: 431244:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 6600.103841] Lustre: Skipped 1 previous similar message [ 6601.323133] LustreError: 432392:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6601.326329] LustreError: 432392:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6601.390308] Lustre: server umount lustre-OST0000 complete [ 6603.715448] LustreError: 432745:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6603.717119] LustreError: 432745:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6603.803098] Lustre: server umount lustre-MDT0001 complete [ 6607.852804] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 6609.028483] Key type lgssc unregistered [ 6609.178528] LNet: 433272:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6609.181195] LNetError: 433272:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6609.189490] LNet: Removed LNI 192.168.203.133@tcp [ 6609.539344] Key type .llcrypt unregistered [ 6609.540952] Key type ._llcrypt unregistered [ 6619.643657] Key type ._llcrypt registered [ 6619.645319] Key type .llcrypt registered [ 6619.686789] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 6620.080517] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6620.094702] alg: No test for adler32 (adler32-zlib) [ 6620.978399] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 6621.076765] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 6622.663212] Key type lgssc registered [ 6623.039635] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6625.894463] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6625.901284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6627.063256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6628.384978] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6631.341547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6631.487044] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6632.820603] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6634.626582] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6635.928302] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6638.267966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6638.386513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6640.283524] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6642.092176] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6652.710190] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6662.111861] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6662.114062] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6662.117690] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6662.915300] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.33@tcp (stopping) [ 6663.137544] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6664.762955] LustreError: 436885:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6664.781782] Lustre: server umount lustre-MDT0001 complete [ 6668.256521] LustreError: 434637:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6668.257228] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6668.257733] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6668.258239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6668.258244] Lustre: Skipped 2 previous similar messages [ 6668.263816] LustreError: 434637:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 6672.557468] LustreError: 437088:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6672.560022] LustreError: 437088:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6672.619662] Lustre: server umount lustre-OST0000 complete [ 6673.377131] LustreError: 434637:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6673.979864] LustreError: 437289:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6673.981825] LustreError: 437289:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6674.072320] Lustre: server umount lustre-MDT0000 complete [ 6678.988454] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 6680.128889] Key type lgssc unregistered [ 6680.279556] LNet: 437967:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6680.282245] LNetError: 437967:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6680.288352] LNet: Removed LNI 192.168.203.133@tcp [ 6680.594475] Key type .llcrypt unregistered [ 6680.595778] Key type ._llcrypt unregistered [ 6688.919470] Key type ._llcrypt registered [ 6688.921123] Key type .llcrypt registered [ 6688.964874] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 6695.228434] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 6695.537596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6695.555173] alg: No test for adler32 (adler32-zlib) [ 6696.405553] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 6696.508401] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 6698.095178] Key type lgssc registered [ 6698.476123] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6701.526850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6703.739385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6705.765792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6707.892197] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6711.205402] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 6714.215557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6716.488424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6716.514842] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6716.520300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6717.622519] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6717.634676] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6717.671562] Lustre: lustre-MDT0000: new disk, initializing [ 6717.699290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6717.706409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6719.084205] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6722.711438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6725.057823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6725.081109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6725.174235] Lustre: lustre-OST0000: new disk, initializing [ 6725.175949] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6725.177506] Lustre: Skipped 1 previous similar message [ 6725.198795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6726.342421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6726.345258] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6726.356222] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6727.194598] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6730.296480] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6733.647222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6735.915467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6735.944744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6735.982694] Lustre: 441893:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6736.023595] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6736.063735] Lustre: lustre-MDT0001: new disk, initializing [ 6736.118619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6736.136642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 6736.140962] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 6737.579124] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6741.264336] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6741.337154] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6742.714048] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6743.806943] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 1 sec [ 6747.399632] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.33@tcp (stopping) [ 6748.129904] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6748.130622] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6748.131202] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6748.134939] Lustre: Skipped 1 previous similar message [ 6751.546657] LustreError: 445289:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6751.569404] Lustre: server umount lustre-MDT0001 complete [ 6758.369622] LustreError: 441898:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6758.377231] LustreError: 441898:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 5 previous similar messages [ 6763.490042] LustreError: 442899:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6763.493825] LustreError: 442899:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 6767.583957] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6767.587591] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6767.593819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6767.596520] Lustre: Skipped 1 previous similar message [ 6768.609531] LustreError: 441904:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6773.728494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6773.728917] LustreError: 441904:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6773.731473] Lustre: Skipped 1 previous similar message [ 6778.335138] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6778.357477] LustreError: 445562:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 6778.359563] LustreError: 445562:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6778.424874] Lustre: server umount lustre-OST0000 complete [ 6778.848743] LustreError: 441904:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6794.207119] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6794.248753] LustreError: 445766:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 6794.251161] LustreError: 445766:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6794.339649] Lustre: server umount lustre-MDT0000 complete [ 6799.077135] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 6800.119959] Key type lgssc unregistered [ 6800.252366] LNet: 446445:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6800.255220] LNetError: 446445:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6800.264345] LNet: Removed LNI 192.168.203.133@tcp [ 6800.551766] Key type .llcrypt unregistered [ 6800.552896] Key type ._llcrypt unregistered [ 6813.642035] Key type ._llcrypt registered [ 6813.643546] Key type .llcrypt registered [ 6813.688522] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 6813.793203] Key type .llcrypt unregistered [ 6813.794236] Key type ._llcrypt unregistered [ 6820.634523] Key type ._llcrypt registered [ 6820.635609] Key type .llcrypt registered [ 6820.676974] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 6821.239327] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6821.253201] alg: No test for adler32 (adler32-zlib) [ 6822.105320] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 6822.184137] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 6823.759130] Key type lgssc registered [ 6824.111310] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6826.855154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6829.006842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6831.025498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6833.192140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6833.217113] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6833.223751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6834.320554] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6834.336112] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6834.381360] Lustre: lustre-MDT0000: new disk, initializing [ 6834.408530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6834.415242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6835.668700] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6839.769207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6839.792428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6839.824019] Lustre: 449751:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6839.845744] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6839.847537] Lustre: Skipped 1 previous similar message [ 6839.877587] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6839.888036] Lustre: lustre-MDT0001: new disk, initializing [ 6839.919270] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6839.932199] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6839.935143] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6841.205820] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6844.340270] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6845.577230] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6847.898397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6847.923444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6848.040760] Lustre: lustre-OST0000: new disk, initializing [ 6848.042619] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6848.078718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6849.955947] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6853.047246] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6855.671545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6855.675888] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6855.706630] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6860.770203] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6860.770513] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6860.771079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6860.775593] Lustre: Skipped 1 previous similar message [ 6865.888746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6865.891146] Lustre: Skipped 4 previous similar messages [ 6866.813132] LustreError: 452177:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6866.840195] Lustre: server umount lustre-MDT0000 complete [ 6868.102054] LustreError: 449743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754412177 with bad export cookie 1566118101149847417 [ 6868.104926] LustreError: 449743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6868.105571] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6868.172162] LustreError: 452379:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6868.173880] LustreError: 452379:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6868.248362] Lustre: server umount lustre-MDT0001 complete [ 6879.729622] LustreError: 452580:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6879.732168] LustreError: 452580:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6879.806268] Lustre: server umount lustre-OST0000 complete [ 6883.271454] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 12:43:12 (1754412192) [ 6883.639787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6885.889198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6886.082476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6887.437774] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6890.289068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6890.409091] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6891.685446] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6893.495457] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6894.689144] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6897.103706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6897.132107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6897.194186] Lustre: Found index 0 for lustre-OST0000, updating log [ 6898.986468] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6900.645402] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6910.570689] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 6916.066853] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6916.069881] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6916.075831] Lustre: Skipped 1 previous similar message [ 6916.078901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6916.081099] Lustre: Skipped 1 previous similar message [ 6918.574080] LustreError: 455664:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6918.575924] LustreError: 455664:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6918.637650] Lustre: server umount lustre-OST0000 complete [ 6919.567435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6921.183787] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6921.185939] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6921.188624] Lustre: Skipped 1 previous similar message [ 6921.189938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6921.191513] Lustre: Skipped 2 previous similar messages [ 6926.268041] LustreError: 455921:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6926.269797] LustreError: 455921:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6926.300781] Lustre: server umount lustre-MDT0000 complete [ 6927.538161] LustreError: 453387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754412237 with bad export cookie 1566118101149848817 [ 6927.540784] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6927.542093] LustreError: 453387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6927.692967] Lustre: server umount lustre-MDT0001 complete [ 6930.479401] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 12:43:59 (1754412239) [ 6930.848729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6931.187594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6931.527445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6931.868718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6932.195083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6932.516289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6932.830417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6933.157528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6933.491882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6933.815440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6934.138378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6934.473806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6934.801027] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6935.118682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6935.454417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6935.781636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6936.108333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6936.442716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6936.779301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6937.131341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6937.505598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6937.831833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6938.147179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6938.483581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6938.832531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6939.178739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6939.552426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6939.917269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6940.252218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6940.596732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6940.935674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6941.278828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6941.644629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6942.002582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6942.373421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6942.715380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6943.048482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6943.387861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6943.735048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6944.079081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6944.442361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6944.795226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6945.139521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6945.494188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6945.846977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6946.196920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6946.548793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6946.892204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6947.248824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6947.594248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6947.937346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6950.601536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6950.755334] LustreError: 459503:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6950.762072] LustreError: 459503:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 6950.781633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6950.783561] Lustre: Skipped 1 previous similar message [ 6952.091380] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6955.312565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6957.154393] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6960.479807] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6963.351866] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6967.618654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6967.683891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6967.816060] Lustre: Found index 0 for lustre-OST0000, updating log [ 6967.963565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6967.966771] Lustre: Skipped 1 previous similar message [ 6971.381845] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 6974.942126] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6976.321276] LustreError: 461758:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6976.323638] LustreError: 461758:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6976.414882] Lustre: server umount lustre-OST0000 complete [ 6981.088338] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6981.091433] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6981.096053] Lustre: Skipped 1 previous similar message [ 6981.098398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6981.100696] Lustre: Skipped 4 previous similar messages [ 6993.375140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6993.409915] LustreError: 461958:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6993.412249] LustreError: 461958:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6993.438663] Lustre: server umount lustre-MDT0000 complete [ 6994.400180] LustreError: 460209:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6994.407136] LustreError: 460209:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 6994.720244] LustreError: 459484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754412304 with bad export cookie 1566118101149849888 [ 6994.724210] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6994.725811] LustreError: 459484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6994.857067] Lustre: server umount lustre-MDT0001 complete [ 6995.800519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7001.076572] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 7004.368332] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 7007.681154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7009.860666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7011.934130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7014.128465] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7016.507837] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 12:45:25 (1754412325) [ 7033.238915] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 7034.311417] Key type lgssc unregistered [ 7034.454773] LNet: 465723:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7034.458125] LNetError: 465723:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7161.445878] LNet: Removed LNI 192.168.203.133@tcp [ 7161.741070] Key type .llcrypt unregistered [ 7161.742141] Key type ._llcrypt unregistered [ 7168.382175] Key type ._llcrypt registered [ 7168.383090] Key type .llcrypt registered [ 7168.416036] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 7168.783311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7168.799080] alg: No test for adler32 (adler32-zlib) [ 7169.643758] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 7169.729044] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 7171.311163] Key type lgssc registered [ 7171.667694] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7173.038422] LNet: Added LNI 192.168.203.133@tcp100 [8/256/0/180] [ 7173.570233] Lustre: DEBUG MARKER: SKIP: conf-sanity test_73c unable to configure net [ 7179.550834] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 7180.748451] Key type lgssc unregistered [ 7180.915311] LNet: 467579:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7180.917953] LNetError: 467579:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7180.920179] LNet: Removed LNI 192.168.203.133@tcp [ 7180.928326] LNet: Removed LNI 192.168.203.133@tcp100 [ 7181.218601] Key type .llcrypt unregistered [ 7181.220097] Key type ._llcrypt unregistered [ 7189.005701] Key type ._llcrypt registered [ 7189.007670] Key type .llcrypt registered [ 7189.048146] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 7195.544051] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 7195.923173] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7195.963227] alg: No test for adler32 (adler32-zlib) [ 7196.825456] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 7196.909859] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 7198.495163] Key type lgssc registered [ 7198.871648] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7202.042609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7204.352772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7206.407629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7208.559522] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7210.833696] Lustre: DEBUG MARKER: == conf-sanity test 74: Test per-device adaptive timeout parameters ========================================================== 12:48:39 (1754412519) [ 7213.021508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7213.045847] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7213.051191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7214.133418] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7214.143740] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7214.179184] Lustre: lustre-MDT0000: new disk, initializing [ 7214.204897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7214.212361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7215.599628] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 7219.748114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7219.772866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7219.805031] Lustre: 471054:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7219.827776] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7219.829369] Lustre: Skipped 1 previous similar message [ 7219.870536] Lustre: lustre-MDT0001: new disk, initializing [ 7219.907326] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7219.918965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7219.923660] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7221.181124] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 7224.293994] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7225.531156] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7227.757753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7227.787988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7227.893212] Lustre: lustre-OST0000: new disk, initializing [ 7227.895279] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7227.926204] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7229.367562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7229.370521] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7229.405064] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7229.919968] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 7233.027751] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7237.590517] loop0: detected capacity change from 0 to 204800000 [ 7237.645371] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7237.650025] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7237.656121] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7237.659646] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7237.679543] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7237.701966] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7237.716466] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7237.739673] blk_update_request: operation not supported error, dev loop0, sector 176912 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7237.769374] blk_update_request: operation not supported error, dev loop0, sector 185112 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7237.798713] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7239.831833] loop0: detected capacity change from 0 to 204800000 [ 7239.890577] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7239.950210] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7241.727215] loop0: detected capacity change from 0 to 204800000 [ 7243.616434] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7243.640867] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7243.693650] Lustre: fs15246-OST0000: new disk, initializing [ 7243.696419] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 7243.715223] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7245.633023] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 7249.555350] loop1: detected capacity change from 0 to 204800000 [ 7251.472961] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7251.508959] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7251.545100] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 7251.573830] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 7251.621789] Lustre: fs15246-MDT0000: new disk, initializing [ 7251.682217] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7251.691716] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7251.739728] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7252.768660] Lustre: 475597:0:(client.c:1611:after_reply()) @@@ resending request on EINPROGRESS req@00000000ed980daf x1839634847929344/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1754412578 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 7253.043517] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 7257.824835] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 7257.827349] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 7257.834393] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 7262.694292] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7262.699373] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 7266.552264] LustreError: 476713:0:(obd_class.h:479:obd_check_dev()) Device 30 not setup [ 7266.580574] Lustre: server umount fs15246-MDT0000 complete [ 7269.710139] LustreError: 477215:0:(obd_class.h:479:obd_check_dev()) Device 32 not setup [ 7269.712012] LustreError: 477215:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7269.727074] Lustre: server umount fs15246-OST0000 complete [ 7275.488138] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7275.488300] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7275.489888] LustreError: Skipped 1 previous similar message [ 7275.490346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7275.493879] Lustre: Skipped 1 previous similar message [ 7275.497167] Lustre: Skipped 1 previous similar message [ 7278.049728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7278.052197] Lustre: Skipped 1 previous similar message [ 7279.281134] LustreError: 477716:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7279.284163] LustreError: 477716:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 7279.357865] Lustre: server umount lustre-OST0000 complete [ 7281.632332] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7281.635913] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7281.641941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7286.792288] LustreError: 477918:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7286.795073] LustreError: 477918:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7286.829680] Lustre: server umount lustre-MDT0000 complete [ 7288.113679] LustreError: 471048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754412597 with bad export cookie 17714057579741596616 [ 7288.117709] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7288.118850] LustreError: 471048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7288.283023] Lustre: server umount lustre-MDT0001 complete [ 7292.018472] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 7293.090738] Key type lgssc unregistered [ 7293.221356] LNet: 478647:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7293.224408] LNetError: 478647:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7293.232344] LNet: Removed LNI 192.168.203.133@tcp [ 7293.548483] Key type .llcrypt unregistered [ 7293.549560] Key type ._llcrypt unregistered [ 7300.733994] Key type ._llcrypt registered [ 7300.734918] Key type .llcrypt registered [ 7300.774709] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 7301.086722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7301.093266] alg: No test for adler32 (adler32-zlib) [ 7301.959972] Lustre: Lustre: Build Version: 2.16.57_28_g7ecc534 [ 7302.050206] LNet: Added LNI 192.168.203.133@tcp [8/256/0/180] [ 7303.639215] Key type lgssc registered [ 7304.047296] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7306.722069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7308.406583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7310.265986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7311.968874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7317.327281] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 7320.454873] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 7324.069773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7326.399804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7328.548516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7330.751984] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7333.115315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7333.134332] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7333.139484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7334.235752] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7334.249833] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7334.292132] Lustre: lustre-MDT0000: new disk, initializing [ 7334.318280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7334.325160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7335.654437] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 7339.882824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7339.911894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7339.953585] Lustre: 483037:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7339.979295] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7339.980935] Lustre: Skipped 1 previous similar message [ 7340.018658] Lustre: lustre-MDT0001: new disk, initializing [ 7340.055556] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7340.068803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7340.071809] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7341.427402] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 7344.603819] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7345.839041] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7348.142719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7348.168418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7348.269703] Lustre: lustre-OST0000: new disk, initializing [ 7348.272092] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7348.302639] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7350.247471] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_default_debug -1 all [ 7350.523113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7350.529210] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7350.568886] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7353.400844] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7354.745177] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7354.819958] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7356.198853] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7356.289217] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7360.995032] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7360.996137] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7360.998428] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7361.003095] Lustre: Skipped 1 previous similar message [ 7366.115798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7366.117913] Lustre: Skipped 2 previous similar messages [ 7371.231087] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7371.233631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7371.235159] Lustre: Skipped 1 previous similar message [ 7371.249097] LustreError: 485939:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7371.311606] Lustre: server umount lustre-OST0000 complete [ 7375.840402] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7375.844206] LustreError: Skipped 1 previous similar message [ 7375.846053] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7375.851101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7378.689846] LustreError: 486142:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7378.692138] LustreError: 486142:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7378.724545] Lustre: server umount lustre-MDT0000 complete [ 7379.981676] LustreError: 483028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754412689 with bad export cookie 9057481960314461984 [ 7379.984681] LustreError: MGC192.168.203.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7379.984932] LustreError: 483028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7380.063306] LustreError: 486343:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7380.066890] LustreError: 486343:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7380.168455] Lustre: server umount lustre-MDT0001 complete [ 7389.265925] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing set_hostid [ 7392.990576] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing load_modules_local [ 7396.926672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7399.960307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7402.329431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7404.705265] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7405.313435] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7244 sec ========== 12:51:54 (1754412714) [ 7405.914965] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:51:54 (1754412714) === [ 7407.238053] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:51:56 (1754412716) === [ 7420.862197] Lustre: DEBUG MARKER: oleg333-server.virtnet: executing unload_modules_local [ 7422.063947] Key type lgssc unregistered [ 7422.215505] LNet: 491175:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7422.218340] LNetError: 491175:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7422.226490] LNet: Removed LNI 192.168.203.133@tcp [ 7422.577390] Key type .llcrypt unregistered [ 7422.578486] Key type ._llcrypt unregistered