[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 412594898 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001024] APIC: Switch to symmetric I/O mode setup [ 0.002497] x2apic enabled [ 0.003004] Switched APIC routing to physical x2apic. [ 0.004011] kvm-guest: setup PV IPIs [ 0.007364] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009009] pid_max: default: 32768 minimum: 301 [ 0.011132] LSM: Security Framework initializing [ 0.012039] Yama: becoming mindful. [ 0.013030] SELinux: Initializing. [ 0.014044] *** VALIDATE selinux *** [ 0.021420] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025446] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026131] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027081] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028106] *** VALIDATE tmpfs *** [ 0.030521] *** VALIDATE proc *** [ 0.031206] *** VALIDATE cgroup *** [ 0.032005] *** VALIDATE cgroup2 *** [ 0.034048] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035125] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037033] Spectre V2 : User space: Vulnerable [ 0.038013] Speculative Store Bypass: Vulnerable [ 0.041051] debug: unmapping init [mem 0xffffffff9ea59000-0xffffffff9ea60fff] [ 0.044152] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045545] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046015] ... version: 2 [ 0.047007] ... bit width: 48 [ 0.048006] ... generic registers: 4 [ 0.049007] ... value mask: 0000ffffffffffff [ 0.050006] ... max period: 00007fffffffffff [ 0.051008] ... fixed-purpose events: 3 [ 0.052007] ... event mask: 000000070000000f [ 0.053350] rcu: Hierarchical SRCU implementation. [ 0.055410] smp: Bringing up secondary CPUs ... [ 0.056476] x86: Booting SMP configuration: [ 0.057027] .... node #0, CPUs: #1 #2 #3 [ 0.060366] smp: Brought up 1 node, 4 CPUs [ 0.062010] smpboot: Max logical packages: 1 [ 0.063014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.224593] node 0 deferred pages initialised in 160ms [ 0.228117] devtmpfs: initialized [ 0.229213] x86/mm: Memory block size: 128MB [ 0.232754] gcov: version magic: 0x41383552 [ 0.235320] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.239130] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.242368] pinctrl core: initialized pinctrl subsystem [ 0.244194] [ 0.244727] ************************************************************* [ 0.247010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.249009] ** ** [ 0.251011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.254010] ** ** [ 0.256010] ** This means that this kernel is built to expose internal ** [ 0.258010] ** IOMMU data structures, which may compromise security on ** [ 0.260010] ** your system. ** [ 0.262011] ** ** [ 0.264009] ** If you see this message and you are not debugging the ** [ 0.266008] ** kernel, report this immediately to your vendor! ** [ 0.268010] ** ** [ 0.270010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.272009] ************************************************************* [ 0.274817] NET: Registered protocol family 16 [ 0.275449] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.278047] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.280049] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.284018] cpuidle: using governor menu [ 0.285624] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.288602] PCI: Using configuration type 1 for base access [ 0.291170] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.300097] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.303015] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.307046] cryptd: max_cpu_qlen set to 1000 [ 0.310330] ACPI: Added _OSI(Module Device) [ 0.313019] ACPI: Added _OSI(Processor Device) [ 0.315022] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.317014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.322485] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.331149] ACPI: Interpreter enabled [ 0.332058] ACPI: PM: (supports S0 S3 S4 S5) [ 0.334016] ACPI: Using IOAPIC for interrupt routing [ 0.336153] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.339474] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.348754] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.351029] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.354024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.356091] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.360381] acpiphp: Slot [2] registered [ 0.362129] acpiphp: Slot [5] registered [ 0.363098] acpiphp: Slot [6] registered [ 0.364091] acpiphp: Slot [7] registered [ 0.365098] acpiphp: Slot [8] registered [ 0.366043] acpiphp: Slot [9] registered [ 0.367000] acpiphp: Slot [10] registered [ 0.367000] acpiphp: Slot [3] registered [ 0.369074] acpiphp: Slot [4] registered [ 0.370063] acpiphp: Slot [11] registered [ 0.372072] acpiphp: Slot [12] registered [ 0.373080] acpiphp: Slot [13] registered [ 0.374056] acpiphp: Slot [14] registered [ 0.375068] acpiphp: Slot [15] registered [ 0.377072] acpiphp: Slot [16] registered [ 0.378088] acpiphp: Slot [17] registered [ 0.379114] acpiphp: Slot [18] registered [ 0.381072] acpiphp: Slot [19] registered [ 0.382061] acpiphp: Slot [20] registered [ 0.383068] acpiphp: Slot [21] registered [ 0.385080] acpiphp: Slot [22] registered [ 0.386087] acpiphp: Slot [23] registered [ 0.387060] acpiphp: Slot [24] registered [ 0.387998] acpiphp: Slot [25] registered [ 0.389050] acpiphp: Slot [26] registered [ 0.390051] acpiphp: Slot [27] registered [ 0.391137] acpiphp: Slot [28] registered [ 0.392078] acpiphp: Slot [29] registered [ 0.394085] acpiphp: Slot [30] registered [ 0.395091] acpiphp: Slot [31] registered [ 0.396049] PCI host bridge to bus 0000:00 [ 0.398013] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.399011] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.402086] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.404020] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.406011] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.409047] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.411244] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.413806] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.416231] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.424805] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.429485] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.431023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.433020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.436022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.439170] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.441908] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.443036] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.449374] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.454012] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.466771] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.472012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.476476] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.483024] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.489031] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.513024] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.521131] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.531018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.538015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.555014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.564639] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.572015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.579020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.602015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.615612] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.625019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.632040] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.652036] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.668214] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.677014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.683012] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.699022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.712558] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.716011] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.721016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.733014] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.739153] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.741327] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.743246] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.744234] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.746355] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.751142] iommu: Default domain type: Passthrough [ 0.752274] SCSI subsystem initialized [ 0.753077] ACPI: bus type USB registered [ 0.754026] usbcore: registered new interface driver usbfs [ 0.755033] usbcore: registered new interface driver hub [ 0.756000] usbcore: registered new device driver usb [ 0.757118] pps_core: LinuxPPS API ver. 1 registered [ 0.759011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.763053] PTP clock support registered [ 0.766423] EDAC MC: Ver: 3.0.0 [ 0.769155] PCI: Using ACPI for IRQ routing [ 0.772803] NetLabel: Initializing [ 0.774011] NetLabel: domain hash size = 128 [ 0.776009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.779082] NetLabel: unlabeled traffic allowed by default [ 0.782065] vgaarb: loaded [ 0.784386] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.785009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.792394] clocksource: Switched to clocksource kvm-clock [ 0.884956] VFS: Disk quotas dquot_6.6.0 [ 0.886356] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.888719] *** VALIDATE ramfs *** [ 0.889813] *** VALIDATE hugetlbfs *** [ 0.892029] pnp: PnP ACPI init [ 0.894150] pnp: PnP ACPI: found 6 devices [ 0.909096] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.912749] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.915042] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.916849] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.919155] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.921316] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.924209] NET: Registered protocol family 2 [ 0.926931] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.930649] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.932570] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.935843] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.938594] TCP: Hash tables configured (established 65536 bind 65536) [ 0.941928] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.944179] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.946184] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.948179] NET: Registered protocol family 1 [ 0.949898] RPC: Registered named UNIX socket transport module. [ 0.952990] RPC: Registered udp transport module. [ 0.954200] RPC: Registered tcp transport module. [ 0.955927] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.957943] NET: Registered protocol family 44 [ 0.959049] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.961309] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.962850] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.965521] PCI: CLS 0 bytes, default 64 [ 0.967757] Unpacking initramfs... [ 2.490649] debug: unmapping init [mem 0xffff8e1a7cc54000-0xffff8e1a7ffbffff] [ 2.495139] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.497306] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.500159] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.295798] Initialise system trusted keyrings [ 3.298506] Key type blacklist registered [ 3.321882] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.375843] zbud: loaded [ 3.380165] *** VALIDATE nfs *** [ 3.381586] *** VALIDATE nfs4 *** [ 3.383401] pstore: using deflate compression [ 3.387941] Platform Keyring initialized [ 3.519551] NET: Registered protocol family 38 [ 3.521252] Key type asymmetric registered [ 3.522815] Asymmetric key parser 'x509' registered [ 3.524850] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.528484] io scheduler mq-deadline registered [ 3.530396] io scheduler kyber registered [ 3.531945] io scheduler bfq registered [ 3.535266] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.538621] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.541633] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.544833] ACPI: Power Button [PWRF] [ 3.552230] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.559873] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.578187] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.586751] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.604504] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.631903] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.663249] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.669244] Non-volatile memory driver v1.3 [ 3.670659] Linux agpgart interface v0.103 [ 3.707230] virtio_blk virtio1: [vda] 145976 512-byte logical blocks (74.7 MB/71.3 MiB) [ 3.710256] vda: detected capacity change from 0 to 74739712 [ 3.725205] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.728679] vdb: detected capacity change from 0 to 1073741824 [ 3.746356] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.750848] vdc: detected capacity change from 0 to 2621440000 [ 3.765671] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.768762] vdd: detected capacity change from 0 to 2621440000 [ 3.784233] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.787306] vde: detected capacity change from 0 to 4294967296 [ 3.802234] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.805228] vdf: detected capacity change from 0 to 4294967296 [ 3.811653] libphy: Fixed MDIO Bus: probed [ 3.817549] usbcore: registered new interface driver usbserial_generic [ 3.820068] usbserial: USB Serial support registered for generic [ 3.822560] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.826664] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.828594] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.831174] mousedev: PS/2 mouse device common for all mice [ 3.834220] rtc_cmos 00:05: RTC can wake from S4 [ 3.835766] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.837408] rtc_cmos 00:05: registered as rtc0 [ 3.837462] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.837500] intel_pstate: CPU model not supported [ 3.841489] hid: raw HID events driver (C) Jiri Kosina [ 3.846117] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.847132] usbcore: registered new interface driver usbhid [ 3.847136] usbhid: USB HID core driver [ 3.847268] drop_monitor: Initializing network drop monitor service [ 3.848499] Initializing XFRM netlink socket [ 3.856172] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.860650] NET: Registered protocol family 10 [ 3.876542] Segment Routing with IPv6 [ 3.877921] NET: Registered protocol family 17 [ 3.879995] mpls_gso: MPLS GSO support [ 3.885867] RAS: Correctable Errors collector initialized. [ 3.888098] AVX version of gcm_enc/dec engaged. [ 3.889829] AES CTR mode by8 optimization enabled [ 3.977789] sched_clock: Marking stable (3977642166, 0)->(4785448780, -807806614) [ 3.981638] registered taskstats version 1 [ 3.983616] Loading compiled-in X.509 certificates [ 3.987436] zswap: loaded using pool lzo/zbud [ 4.013749] Key type big_key registered [ 4.026686] Key type encrypted registered [ 4.028480] ima: No TPM chip found, activating TPM-bypass! [ 4.031659] ima: Allocated hash algorithm: sha1 [ 4.033609] ima: No architecture policies found [ 4.035767] evm: Initialising EVM extended attributes: [ 4.041238] evm: security.selinux [ 4.042298] evm: security.ima [ 4.043223] evm: security.capability [ 4.044696] evm: HMAC attrs: 0x1 [ 4.047196] rtc_cmos 00:05: setting system clock to 2026-08-11 20:19:18 UTC (1786479558) [ 4.054837] debug: unmapping init [mem 0xffffffff9fa03000-0xffffffff9fbfffff] [ 4.058395] debug: unmapping init [mem 0xffffffff9e782000-0xffffffff9ea58fff] [ 4.067391] Write protecting the kernel read-only data: 28672k [ 4.071533] debug: unmapping init [mem 0xffffffff9ce03000-0xffffffff9cffffff] [ 4.074507] debug: unmapping init [mem 0xffffffff9d714000-0xffffffff9d7fffff] [ 4.114671] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.125678] systemd[1]: Detected virtualization kvm. [ 4.127884] systemd[1]: Detected architecture x86-64. [ 4.130163] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.161974] systemd[1]: No hostname configured. [ 4.204046] systemd[1]: Set hostname to . [ 4.206539] random: systemd: uninitialized urandom read (16 bytes read) [ 4.209433] systemd[1]: Initializing machine ID from random generator. [ 4.431547] random: ln: uninitialized urandom read (6 bytes read) [ 4.732787] random: systemd: uninitialized urandom read (16 bytes read) [ 4.736615] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 4.747960] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.755624] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Reached target Timers. Starting Setup Virtual Console... [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.665258] device-mapper: uevent: version 1.0.3 [ 5.667593] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 7.124099] virtio_net virtio0 ens2: renamed from eth0 [ 7.222959] random: fast init done [ 7.669471] scsi host0: ata_piix [ 7.689393] scsi host1: ata_piix [ 7.692113] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.701116] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.556859] random: crng init done [ 13.558327] random: 7 urandom warning(s) missed due to ratelimiting [ 16.281257] dracut-initqueue[593]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 17.061480] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 18.506965] printk: systemd: 19 output lines suppressed due to ratelimiting [ 18.910732] SELinux: Disabled at runtime. [ 18.999585] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 19.011203] systemd[1]: Detected virtualization kvm. [ 19.014295] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 19.840302] systemd[1]: initrd-switch-root.service: Succeeded. [ 19.843676] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 19.849742] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 19.855529] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 19.859779] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 19.871170] systemd[1]: Starting Journal Service... Starting Journal Service... [ 19.882330] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Initrd File Systems. Starting Remount Root and Kernel File Systems... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Mounting POSIX Message Queue File System... Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-getty.slice. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System.[ 20.312156] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 20.661357] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 21.364871] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 21.473908] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 21.854840] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 22.074237] EDAC sbridge: Ver: 1.1.2 [ 24.508866] Key type dns_resolver registered [ 24.990254] NFS: Registering the id_resolver key type [ 24.992706] Key type id_resolver registered [ 24.994537] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Login Service. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg322-server login: [ 66.009969] libcfs: loading out-of-tree module taints kernel. [ 66.028412] Key type ._llcrypt registered [ 66.030150] Key type .llcrypt registered [ 66.071571] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 74.133518] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 74.707183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 74.712958] alg: No test for adler32 (adler32-zlib) [ 75.704230] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 76.007859] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 77.624132] Key type lgssc registered [ 78.219060] Lustre: Echo OBD driver; http://www.lustre.org/ [ 85.505622] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 102.979801] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 108.440412] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 108.454242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 109.567469] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 109.582609] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 109.629959] Lustre: lustre-MDT0000: new disk, initializing [ 109.672876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 109.683562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 111.384689] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 117.722395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 117.772787] Lustre: 6489:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 117.791406] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 117.794868] Lustre: Skipped 1 previous similar message [ 117.837607] Lustre: lustre-MDT0001: new disk, initializing [ 117.870134] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 117.883821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 117.888090] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 119.473377] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 122.060681] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 125.787268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 125.889537] Lustre: lustre-OST0000: new disk, initializing [ 125.892105] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 125.895353] Lustre: 8427:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 125.917144] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 128.133377] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 132.595288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 132.600069] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 132.632934] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 134.253741] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 134.310598] Lustre: lustre-OST0001: new disk, initializing [ 134.312322] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 134.315231] Lustre: 9501:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 134.341038] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 137.079857] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 137.852652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 137.867860] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 137.894057] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 143.836166] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 150.806234] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 153.457126] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing check_logdir /tmp/testlogs/ [ 155.450306] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing yml_node [ 157.416961] Lustre: DEBUG MARKER: Client: 2.17.56.51 [ 158.443158] Lustre: DEBUG MARKER: MDS: 2.17.56.51 [ 159.539205] Lustre: DEBUG MARKER: OSS: 2.17.56.51 [ 160.263684] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Aug 11 16:21:54 EDT 2026 [ 167.014790] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 167.613290] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 184.288963] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 184.289370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 184.294915] Lustre: Skipped 3 previous similar messages [ 184.301447] Lustre: Skipped 3 previous similar messages [ 187.100462] Lustre: server umount lustre-MDT0000 complete [ 189.957835] LustreError: 6483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786479744 with bad export cookie 16823323813616763719 [ 189.962770] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 189.970386] LustreError: 6483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 194.528968] LustreError: 9041:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 194.530297] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 194.539279] LustreError: 9041:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 194.550144] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 194.553311] Lustre: Skipped 1 previous similar message [ 199.653264] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 199.653615] LustreError: 6497:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 199.659323] Lustre: Skipped 1 previous similar message [ 199.673115] LustreError: 6497:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 200.676406] LustreError: 6498:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 200.686952] LustreError: 6498:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 204.256113] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 204.423259] Lustre: server umount lustre-MDT0001 complete [ 208.493846] Lustre: server umount lustre-OST0000 complete [ 211.191628] Lustre: server umount lustre-OST0001 complete [ 216.876586] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 226.209825] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 254.082061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 254.281837] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 254.299819] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 254.350288] Lustre: lustre-MDT0000: new disk, initializing [ 254.414712] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 254.434125] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 258.211316] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 266.783313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 266.863124] Lustre: 15048:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 266.896932] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 266.901725] Lustre: Skipped 1 previous similar message [ 266.954404] Lustre: lustre-MDT0001: new disk, initializing [ 267.017159] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 267.094663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 267.105741] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 270.558429] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 277.967390] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 282.217353] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 286.827437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 287.016639] Lustre: lustre-OST0000: new disk, initializing [ 287.019931] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 287.024941] Lustre: 17015:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 287.103259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 288.287908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 288.297893] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 288.413135] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 291.823057] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 299.027150] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 302.308357] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 302.492646] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 306.010631] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 306.190214] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 308.709045] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 308.710282] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 308.713901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 308.713909] Lustre: Skipped 2 previous similar messages [ 308.717508] Lustre: Skipped 2 previous similar messages [ 313.831575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 313.842528] Lustre: Skipped 2 previous similar messages [ 314.018585] Lustre: server umount lustre-OST0000 complete [ 318.467746] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 318.476406] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 322.987165] Lustre: server umount lustre-MDT0000 complete [ 324.065936] LustreError: 15060:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 324.080525] LustreError: 15060:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 325.519734] LustreError: 15040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786479879 with bad export cookie 16823323813616765259 [ 325.523378] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 325.528303] LustreError: 15040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 325.771446] Lustre: server umount lustre-MDT0001 complete [ 329.085475] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 16:24:42 (1786479882) [ 330.911614] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 332.245438] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 16:24:45 (1786479885) [ 337.487860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 337.741178] LustreError: 19422:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 337.798300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 340.980660] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 347.229031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 350.901368] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 356.254627] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 360.746390] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 365.337979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 370.052766] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 375.461506] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 392.190470] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 392.195486] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 392.202530] Lustre: Skipped 1 previous similar message [ 392.206559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 392.209480] Lustre: Skipped 2 previous similar messages [ 394.215504] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 398.190291] Lustre: server umount lustre-OST0000 complete [ 403.938616] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 403.943497] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 403.951149] Lustre: Skipped 1 previous similar message [ 407.455719] Lustre: server umount lustre-MDT0000 complete [ 409.582161] LustreError: 20149:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 409.618652] LustreError: 20149:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 410.584917] LustreError: 19404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786479965 with bad export cookie 16823323813616766141 [ 410.607744] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 410.612791] LustreError: 19404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 410.841731] hrtimer: interrupt took 3478381 ns [ 410.956284] Lustre: server umount lustre-MDT0001 complete [ 423.439307] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 425.874731] Key type lgssc unregistered [ 426.122667] LNet: 22977:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 426.126650] LNetError: 22977:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 426.147350] LNet: Removed LNI 192.168.203.122@tcp [ 427.052134] Key type .llcrypt unregistered [ 427.068721] Key type ._llcrypt unregistered [ 458.515419] Key type ._llcrypt registered [ 458.525305] Key type .llcrypt registered [ 458.670714] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 459.888349] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 459.911071] alg: No test for adler32 (adler32-zlib) [ 461.376549] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 461.569707] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 463.288238] Key type lgssc registered [ 465.118944] Lustre: Echo OBD driver; http://www.lustre.org/ [ 474.132114] loop: module loaded [ 474.235326] loop0: detected capacity change from 0 to 204800000 [ 474.495045] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 474.532234] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 475.103218] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 482.197326] loop0: detected capacity change from 0 to 204800000 [ 482.484642] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 482.510632] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 482.550419] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 488.727214] loop0: detected capacity change from 0 to 204800000 [ 494.820655] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 494.848797] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 496.095796] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 496.133658] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 496.196936] Lustre: lustre-MDT0000: new disk, initializing [ 496.270936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 496.282386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 501.794691] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 510.764592] loop1: detected capacity change from 0 to 204800000 [ 516.524705] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 516.736402] Lustre: lustre-OST07c6: new disk, initializing [ 516.739965] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 516.743430] Lustre: Skipped 1 previous similar message [ 516.748440] Lustre: 26164:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 516.875573] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 522.797906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 522.810107] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 522.838084] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 523.919351] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 529.532537] Lustre: 25151:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 255 < left 264, rollback = 2 [ 529.550226] Lustre: 25151:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 529.565126] Lustre: 25151:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 529.576383] Lustre: 25151:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 529.582071] Lustre: 25151:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 529.586835] Lustre: 25151:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 543.686235] Lustre: 25150:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 543.696861] Lustre: 25150:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 543.702849] Lustre: 25150:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 543.713042] Lustre: 25150:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 543.720805] Lustre: 25150:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 543.740585] Lustre: 25150:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 548.863528] Lustre: 25150:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 548.872232] Lustre: 25150:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 548.878540] Lustre: 25150:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 548.885097] Lustre: 25150:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 548.890753] Lustre: 25150:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 548.897200] Lustre: 25150:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 548.903371] Lustre: 25150:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 548.913246] Lustre: 25150:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 548.924350] Lustre: 25150:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 548.932957] Lustre: 25150:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 548.939363] Lustre: 25150:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 548.944781] Lustre: 25150:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 552.271846] Lustre: 25151:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 552.288375] Lustre: 25151:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 552.293963] Lustre: 25151:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 552.304270] Lustre: 25151:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 552.319903] Lustre: 25151:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 552.334533] Lustre: 25151:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 552.352052] Lustre: 25151:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 552.362987] Lustre: 25151:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 552.369292] Lustre: 25151:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 552.377732] Lustre: 25151:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 552.381341] Lustre: 25151:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 552.388667] Lustre: 25151:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 557.652725] Lustre: 26172:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 557.661857] Lustre: 26172:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 557.698559] Lustre: 26172:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 557.710907] Lustre: 26172:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 557.718985] Lustre: 26172:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 557.727576] Lustre: 26172:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 557.735992] Lustre: 26172:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 557.743436] Lustre: 26172:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 557.750691] Lustre: 26172:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 557.760996] Lustre: 26172:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 557.767950] Lustre: 26172:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 557.772537] Lustre: 26172:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 566.306646] Lustre: 25151:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 566.327581] Lustre: 25151:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 566.338102] Lustre: 25151:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 566.349078] Lustre: 25151:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 566.359372] Lustre: 25151:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 566.369834] Lustre: 25151:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 566.380155] Lustre: 25151:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 566.391726] Lustre: 25151:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 566.396986] Lustre: 25151:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 566.402662] Lustre: 25151:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 566.410235] Lustre: 25151:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 566.419056] Lustre: 25151:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 582.682983] Lustre: 25151:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 582.688092] Lustre: 25151:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 7 previous similar messages [ 582.692189] Lustre: 25151:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 582.696654] Lustre: 25151:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 582.701832] Lustre: 25151:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 582.706887] Lustre: 25151:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 582.711778] Lustre: 25151:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 582.716525] Lustre: 25151:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 582.720933] Lustre: 25151:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 582.725790] Lustre: 25151:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 582.730587] Lustre: 25151:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 582.734938] Lustre: 25151:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 620.002609] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 620.002762] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 620.025281] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 624.556849] Lustre: server umount lustre-OST07c6 complete [ 634.632587] Lustre: server umount lustre-MDT0000 complete [ 651.405691] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 653.805512] Key type lgssc unregistered [ 654.190742] LNet: 28721:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 654.203347] LNetError: 28721:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 654.213665] LNet: Removed LNI 192.168.203.122@tcp [ 655.079614] Key type .llcrypt unregistered [ 655.082662] Key type ._llcrypt unregistered [ 672.966687] Key type ._llcrypt registered [ 672.969126] Key type .llcrypt registered [ 673.117705] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 674.391156] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 674.490153] alg: No test for adler32 (adler32-zlib) [ 675.751364] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 676.123226] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 677.864535] Key type lgssc registered [ 678.736358] Lustre: Echo OBD driver; http://www.lustre.org/ [ 685.664357] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 685.678369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 686.995272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 691.797617] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 697.323924] LustreError: 29991:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 697.351866] LustreError: 29991:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 700.436673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 700.669721] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 704.862909] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 712.013390] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 717.675644] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 724.468706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 724.744312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 730.233808] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 731.891552] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 737.953479] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 756.707901] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 756.716725] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 756.729641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 757.402120] Lustre: server umount lustre-OST0000 complete [ 762.341872] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 762.352533] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 762.373091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 767.970113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 767.978819] Lustre: Skipped 2 previous similar messages [ 772.104304] Lustre: server umount lustre-MDT0000 complete [ 773.092193] LustreError: 31240:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 773.105681] LustreError: 31240:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 775.619830] LustreError: 29971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786480330 with bad export cookie 3525826421939227675 [ 775.635164] LustreError: 29971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 775.638325] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 775.999534] Lustre: server umount lustre-MDT0001 complete [ 787.090747] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 789.324499] Key type lgssc unregistered [ 789.680692] LNet: 33730:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 789.685348] LNetError: 33730:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 789.695782] LNet: Removed LNI 192.168.203.122@tcp [ 790.425604] Key type .llcrypt unregistered [ 790.429222] Key type ._llcrypt unregistered [ 809.628976] Key type ._llcrypt registered [ 809.631084] Key type .llcrypt registered [ 809.745405] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 810.915994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 811.091698] alg: No test for adler32 (adler32-zlib) [ 812.285672] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 812.478589] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 814.120259] Key type lgssc registered [ 815.294667] Lustre: Echo OBD driver; http://www.lustre.org/ [ 822.366892] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 822.382366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 823.845036] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 828.014623] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 834.023789] LustreError: 35000:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 834.042764] LustreError: 35000:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 836.153592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 836.399441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 840.555372] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 848.286849] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 854.502206] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 861.092539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 861.156422] LustreError: 36820:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 861.311064] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 866.760905] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 868.864050] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 874.611901] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 884.195609] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 884.201237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 884.210243] Lustre: Skipped 1 previous similar message [ 884.212978] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 884.227591] Lustre: Skipped 1 previous similar message [ 889.326050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 889.344779] Lustre: Skipped 1 previous similar message [ 889.559192] Lustre: server umount lustre-OST0000 complete [ 893.413646] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 893.422731] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 893.449116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 894.435299] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 899.553921] LustreError: 35001:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 899.585071] LustreError: 35001:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 901.316409] Lustre: server umount lustre-MDT0000 complete [ 904.678492] LustreError: 35000:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 904.711850] LustreError: 35000:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 905.012208] LustreError: 34982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786480459 with bad export cookie 11200909124679654296 [ 905.042767] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 905.403327] Lustre: server umount lustre-MDT0001 complete [ 917.431760] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 919.824982] Key type lgssc unregistered [ 920.172706] LNet: 38538:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 920.177838] LNetError: 38538:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 920.186908] LNet: Removed LNI 192.168.203.122@tcp [ 921.189544] Key type .llcrypt unregistered [ 921.191941] Key type ._llcrypt unregistered [ 944.832754] Key type ._llcrypt registered [ 944.834944] Key type .llcrypt registered [ 944.908297] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 945.777755] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 945.792696] alg: No test for adler32 (adler32-zlib) [ 946.926772] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 947.104206] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 948.752180] Key type lgssc registered [ 949.707165] Lustre: Echo OBD driver; http://www.lustre.org/ [ 956.324955] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 956.336739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 957.672896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 961.151334] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 968.163811] LustreError: 39956:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 968.174409] LustreError: 39956:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 969.246302] LustreError: 39957:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 969.996698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 970.320332] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 974.033520] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 980.266092] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 985.297918] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 991.684982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 991.920325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 997.662937] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1004.977342] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1005.244321] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1012.544036] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.22@tcp (stopping) [ 1015.784626] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1015.787394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1015.796938] Lustre: Skipped 2 previous similar messages [ 1015.807977] Lustre: Skipped 2 previous similar messages [ 1017.626423] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.22@tcp (stopping) [ 1018.315608] Lustre: server umount lustre-MDT0000 complete [ 1020.900544] LustreError: 39956:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1020.919235] LustreError: 39956:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1021.787590] LustreError: 39936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786480576 with bad export cookie 10747018474727723871 [ 1021.805978] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1021.808934] LustreError: 39936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1022.797253] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.22@tcp (stopping) [ 1026.020214] LustreError: 39952:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1026.032620] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1026.042256] LustreError: 39952:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1027.860703] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.22@tcp (stopping) [ 1027.864463] Lustre: Skipped 1 previous similar message [ 1031.137663] LustreError: 39951:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1031.149957] LustreError: 39951:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1036.256222] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1036.486723] Lustre: server umount lustre-MDT0001 complete [ 1050.603037] Lustre: server umount lustre-OST0000 complete [ 1063.808830] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 1065.983789] Key type lgssc unregistered [ 1066.284871] LNet: 43819:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1066.289797] LNetError: 43819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1066.300878] LNet: Removed LNI 192.168.203.122@tcp [ 1067.527778] Key type .llcrypt unregistered [ 1067.533337] Key type ._llcrypt unregistered [ 1089.501827] Key type ._llcrypt registered [ 1089.503954] Key type .llcrypt registered [ 1089.618094] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 1090.784256] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1090.803152] alg: No test for adler32 (adler32-zlib) [ 1091.822970] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 1092.029308] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 1093.696151] Key type lgssc registered [ 1094.763903] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1101.436149] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1101.447594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1102.753961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1106.398741] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1113.057739] LustreError: 45235:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1113.070891] LustreError: 45235:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1113.512280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1113.744437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1117.509526] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1123.990075] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1129.996614] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1137.008690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1137.270813] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1142.213874] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1148.187787] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1148.526334] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1155.076871] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1155.092202] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1155.112177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1155.886625] Lustre: lustre-OST0000: Not available for connect from 192.168.203.22@tcp (stopping) [ 1159.143914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1159.144720] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1159.145417] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1160.501434] Lustre: server umount lustre-OST0000 complete [ 1179.621220] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1179.636332] Lustre: Skipped 1 previous similar message [ 1179.644764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1179.653701] Lustre: Skipped 2 previous similar messages [ 1183.719544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1183.793790] Lustre: Skipped 1 previous similar message [ 1191.392228] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1191.594710] Lustre: server umount lustre-MDT0000 complete [ 1193.955192] LustreError: 45232:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1193.964315] LustreError: 45232:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1194.576162] LustreError: 45217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786480749 with bad export cookie 12331948222814230553 [ 1194.581764] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1194.581973] LustreError: 45217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1200.982661] Lustre: server umount lustre-MDT0001 complete [ 1210.212431] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 1212.625202] Key type lgssc unregistered [ 1212.961765] LNet: 48945:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1212.969922] LNetError: 48945:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1213.990658] LNet: Removed LNI 192.168.203.122@tcp [ 1214.742824] Key type .llcrypt unregistered [ 1214.744849] Key type ._llcrypt unregistered [ 1237.811397] Key type ._llcrypt registered [ 1237.813885] Key type .llcrypt registered [ 1237.889501] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 1238.696087] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1238.712774] alg: No test for adler32 (adler32-zlib) [ 1239.691716] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 1239.845891] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 1241.520156] Key type lgssc registered [ 1242.370749] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1248.225286] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1248.239291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1249.513681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1252.652721] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1258.670270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1258.931900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1262.514533] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1268.676048] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1273.070288] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1277.874539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1278.082973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1282.383698] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1287.767258] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1295.477100] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1303.054509] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1305.741324] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1317.657100] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-1116h41m10s [ 1318.717892] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1321.445604] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1321.446884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1321.464180] Lustre: Skipped 2 previous similar messages [ 1321.488760] Lustre: Skipped 2 previous similar messages [ 1322.533422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1322.537723] Lustre: Skipped 1 previous similar message [ 1325.347555] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.22@tcp (stopping) [ 1326.792805] Lustre: server umount lustre-MDT0000 complete [ 1328.612614] LustreError: 50362:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1328.622794] LustreError: 50362:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1329.439974] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1331.529905] LustreError: 50356:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1334.757152] LustreError: 50361:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1334.785621] LustreError: 50361:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1334.914589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1334.993182] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1335.166500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1335.207392] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1336.239707] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1338.473351] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1341.223271] Lustre: DEBUG MARKER: Wait for df (36680) ... [ 1342.925664] Lustre: DEBUG MARKER: done [ 1346.532612] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1346.550021] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1346.556089] Lustre: Skipped 1 previous similar message [ 1351.137453] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1351.142199] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1351.151633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1352.378285] Lustre: server umount lustre-OST0000 complete [ 1356.769482] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1356.777483] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1361.202803] Lustre: server umount lustre-MDT0000 complete [ 1362.914810] LustreError: 51089:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1362.928719] LustreError: 51089:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1364.762327] LustreError: 52204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786480919 with bad export cookie 15860435749848537210 [ 1364.766655] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1364.771719] LustreError: 52204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1364.988242] Lustre: server umount lustre-MDT0001 complete [ 1373.409173] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 1375.474394] Key type lgssc unregistered [ 1375.863512] LNet: 55308:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1375.884482] LNetError: 55308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1376.935817] LNet: Removed LNI 192.168.203.122@tcp [ 1377.527454] Key type .llcrypt unregistered [ 1377.529701] Key type ._llcrypt unregistered [ 1407.842799] Key type ._llcrypt registered [ 1407.844940] Key type .llcrypt registered [ 1407.923668] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 1409.689427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1409.702318] alg: No test for adler32 (adler32-zlib) [ 1410.747306] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 1410.985585] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 1412.667419] Key type lgssc registered [ 1413.834365] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1419.684638] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1419.696825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1420.866517] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1420.883853] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1421.013654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1424.501212] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1431.010967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1431.076527] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1431.103369] Lustre: 57496:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1431.301292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1435.193932] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1441.753407] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1446.513302] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1452.448549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1452.577586] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1452.727154] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1457.905728] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1459.899260] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1464.003730] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1470.315730] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-1116h43m43s [ 1471.633202] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1477.867790] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1479.893278] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1479.951957] LustreError: 57501:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1500.458590] Lustre: lustre-MDT0000: Client 2c5c42a8-53d1-4df7-98fe-3528f3946b37 (at 192.168.203.22@tcp) reconnecting [ 1519.992193] LustreError: 57501:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1521.600741] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1523.455938] Lustre: DEBUG MARKER: done [ 1531.875195] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1531.880675] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1531.892535] LustreError: Skipped 1 previous similar message [ 1531.912249] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1533.835441] Lustre: server umount lustre-OST0000 complete [ 1536.996434] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1537.001239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1537.002874] Lustre: Skipped 2 previous similar messages [ 1537.012478] Lustre: Skipped 1 previous similar message [ 1538.084741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1538.093242] Lustre: Skipped 1 previous similar message [ 1542.116364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1542.120603] Lustre: Skipped 1 previous similar message [ 1542.876391] Lustre: server umount lustre-MDT0000 complete [ 1546.050408] LustreError: 57487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786481100 with bad export cookie 6661141954741264836 [ 1546.063618] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1546.499603] Lustre: server umount lustre-MDT0001 complete [ 1554.959265] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 1556.870981] Key type lgssc unregistered [ 1557.081100] LNet: 61833:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1557.089134] LNetError: 61833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1557.106926] LNet: Removed LNI 192.168.203.122@tcp [ 1557.830485] Key type .llcrypt unregistered [ 1557.832723] Key type ._llcrypt unregistered [ 1590.651944] Key type ._llcrypt registered [ 1590.655176] Key type .llcrypt registered [ 1590.735491] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 1591.577069] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1591.602119] alg: No test for adler32 (adler32-zlib) [ 1592.567439] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 1592.773697] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 1594.416188] Key type lgssc registered [ 1595.321680] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1597.703773] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1597.716439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1598.845463] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1598.858725] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1598.955278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1599.609634] Lustre: Failing over lustre-MDT0000 [ 1599.771355] Lustre: server umount lustre-MDT0000 complete [ 1605.017512] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 16:45:58 (1786481158) [ 1609.706621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1610.036607] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1613.190104] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1619.133660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1619.201674] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1619.226963] Lustre: 64607:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1619.462742] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1622.524221] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1627.765778] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1632.056591] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1637.121160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1637.263222] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1637.427754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1642.752585] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1643.135919] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1649.786900] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1656.397051] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1663.970592] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1664.011898] Lustre: Skipped 2 previous similar messages [ 1665.081562] Lustre: server umount lustre-MDT0000 complete [ 1669.048802] LustreError: 64600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786481223 with bad export cookie 499939369603210474 [ 1669.061270] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1669.070941] LustreError: 64600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1669.096851] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1669.101333] LustreError: 64614:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1669.106227] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1669.118813] LustreError: 64614:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1669.325548] Lustre: server umount lustre-MDT0001 complete [ 1673.546906] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1678.893347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1687.758543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1693.896240] LustreError: 66454:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1693.966598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1694.044299] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1695.022548] LustreError: 66455:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1698.028332] LustreError: 66454:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1698.897590] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1703.398556] LustreError: 66455:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1707.033780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1707.371087] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1710.502753] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1711.348202] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1718.834436] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1719.635823] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1725.601581] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1733.089826] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1733.094711] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1733.102049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1733.513520] Lustre: server umount lustre-MDT0000 complete [ 1734.153294] LustreError: 69814:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1734.202636] LustreError: 69814:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1737.369857] LustreError: 66460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786481291 with bad export cookie 499939369603212756 [ 1737.374113] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1737.386283] LustreError: 66460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1737.625574] Lustre: server umount lustre-MDT0001 complete [ 1744.108327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1753.064129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1754.592644] Lustre: 63536:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786481293/real 1786481293] req@00000000fb81624a x1873261255260416/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786481309 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1754.611755] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1754.621354] Lustre: Skipped 2 previous similar messages [ 1757.664190] Lustre: 63535:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786481296/real 1786481296] req@00000000c6c5700b x1873261255260544/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786481312 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1759.712325] Lustre: 63533:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786481298/real 1786481298] req@00000000c662ab0b x1873261255260672/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786481314 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1761.843771] LustreError: 63532:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000c662ab0b x1873261255261568/t0(0) o250->MGC192.168.203.122@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1762.283760] LustreError: 66454:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1762.466747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1762.574731] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1762.832148] Lustre: 63533:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786481301/real 1786481301] req@000000007d9d3713 x1873261255260928/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786481317 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1766.768599] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1767.844282] Lustre: 63535:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786481306/real 1786481306] req@00000000e021901c x1873261255261056/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786481322 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1774.054896] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1774.732166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1779.517661] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1780.206931] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1788.239965] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1793.895890] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1800.676994] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1800.707025] Lustre: Skipped 2 previous similar messages [ 1800.714474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1800.719122] Lustre: Skipped 1 previous similar message [ 1800.844476] Lustre: server umount lustre-MDT0000 complete [ 1805.124947] LustreError: 66460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786481359 with bad export cookie 499939369603214135 [ 1805.146375] LustreError: 66460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1805.148978] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1805.431438] Lustre: server umount lustre-MDT0001 complete [ 1812.775629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1815.997984] Lustre: DEBUG MARKER: files compared the same [ 1828.349732] Lustre: server umount lustre-OST0000 complete [ 1847.275840] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 1849.269754] Key type lgssc unregistered [ 1849.489669] LNet: 74335:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1849.502931] LNetError: 74335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1849.531666] LNet: Removed LNI 192.168.203.122@tcp [ 1850.337325] Key type .llcrypt unregistered [ 1850.339359] Key type ._llcrypt unregistered [ 1875.915771] Key type ._llcrypt registered [ 1875.917870] Key type .llcrypt registered [ 1876.050417] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 1877.593215] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1877.793484] alg: No test for adler32 (adler32-zlib) [ 1878.798493] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 1879.014797] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 1880.704411] Key type lgssc registered [ 1881.611570] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1888.584877] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1888.597706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1890.094271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1894.198132] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug +malloc all [ 1900.526227] LustreError: 75754:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1900.553359] LustreError: 75754:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1901.986339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1902.189192] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1906.022789] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug +malloc all [ 1914.015375] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1919.843412] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1926.529565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1926.805298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1932.173431] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug +malloc all [ 1935.103882] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1940.112223] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1948.645455] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1948.656427] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1948.675262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1949.518927] Lustre: server umount lustre-OST0000 complete [ 1953.249816] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1953.265876] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1953.283781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1955.822251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1955.822695] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1960.929204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1960.934485] Lustre: Skipped 1 previous similar message [ 1966.054367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1966.064722] Lustre: Skipped 1 previous similar message [ 1967.072279] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1967.205712] Lustre: server umount lustre-MDT0000 complete [ 1970.886604] LustreError: 75733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786481525 with bad export cookie 7740391830441788894 [ 1970.895607] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1970.902363] LustreError: 75733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1971.244466] Lustre: server umount lustre-MDT0001 complete [ 1984.735646] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 1987.720266] Key type lgssc unregistered [ 1988.045977] LNet: 79286:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1988.051366] LNetError: 79286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1988.072776] LNet: Removed LNI 192.168.203.122@tcp [ 1988.806330] Key type .llcrypt unregistered [ 1988.810680] Key type ._llcrypt unregistered [ 2011.365071] Key type ._llcrypt registered [ 2011.382948] Key type .llcrypt registered [ 2011.578667] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 2013.508500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2013.518371] alg: No test for adler32 (adler32-zlib) [ 2014.552267] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 2014.896780] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 2016.552183] Key type lgssc registered [ 2017.718367] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2025.099159] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2025.128418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2040.673191] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2045.792612] LustreError: 80656:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.122@tcp: failed processing log, type 4: rc = -110 [ 2072.683550] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2078.782658] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2085.418244] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2093.547755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2093.611711] Lustre: *** cfs_fail_loc=706, val=0*** [ 2094.518534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2094.681558] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2099.685944] LustreError: 80681:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2099.701780] LustreError: 80681:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2100.201423] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2104.803642] LustreError: 80682:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2106.859214] LustreError: 80681:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2106.875403] LustreError: 80681:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2108.652783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2109.041175] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2113.515184] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2120.928668] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2126.949498] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2130.427661] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2130.431565] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2130.438717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2134.499793] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2134.505596] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2134.506169] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2134.551851] Lustre: Skipped 1 previous similar message [ 2135.806352] Lustre: server umount lustre-OST0000 complete [ 2144.737962] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2144.739783] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2144.789821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2144.798589] Lustre: Skipped 1 previous similar message [ 2146.175831] Lustre: server umount lustre-MDT0000 complete [ 2149.861318] LustreError: 81746:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2149.871632] LustreError: 81746:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2151.200345] LustreError: 80664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786481705 with bad export cookie 1719681795429002978 [ 2151.203118] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2151.224990] LustreError: 80664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2151.508549] Lustre: server umount lustre-MDT0001 complete [ 2163.301931] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 2165.854416] Key type lgssc unregistered [ 2166.061907] LNet: 84277:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2166.068314] LNetError: 84277:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2166.098785] LNet: Removed LNI 192.168.203.122@tcp [ 2166.895856] Key type .llcrypt unregistered [ 2166.900654] Key type ._llcrypt unregistered [ 2192.011661] Key type ._llcrypt registered [ 2192.017859] Key type .llcrypt registered [ 2192.097781] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 2193.589430] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2193.600980] alg: No test for adler32 (adler32-zlib) [ 2194.566212] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 2194.752791] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 2196.376225] Key type lgssc registered [ 2198.244780] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2206.486682] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2206.506165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2211.569917] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2219.671555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2220.240507] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2224.616534] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2225.635423] LustreError: 86383:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2225.651699] LustreError: 86383:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2229.749306] LustreError: 86381:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2229.806244] LustreError: 86381:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2232.856582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2233.065444] LustreError: 86383:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2233.091073] LustreError: 86383:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2233.105906] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2238.442209] LustreError: 86386:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2238.453020] LustreError: 86386:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2238.789958] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2243.559437] LustreError: 86387:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2243.576501] LustreError: 86387:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2248.033082] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2248.044372] Lustre: MGS: Client b4682fc7-55b3-4bb7-a76a-88055841ccd3 (at 0@lo) reconnecting [ 2248.062982] LustreError: 86397:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2248.243313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2248.330892] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2252.768872] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2268.667795] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2268.673128] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2268.682671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2273.755671] Lustre: server umount lustre-OST0000 complete [ 2278.882608] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2278.883464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2278.884240] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2278.889442] Lustre: Skipped 2 previous similar messages [ 2278.892807] Lustre: Skipped 1 previous similar message [ 2284.001391] LustreError: 86381:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2284.008294] LustreError: 86381:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2284.014713] Lustre: server umount lustre-MDT0000 complete [ 2288.054744] Lustre: server umount lustre-MDT0001 complete [ 2291.359603] Lustre: server umount MGS complete [ 2301.792634] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 16:57:35 (1786481855) [ 2327.609511] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 2336.509707] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 2369.186166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2372.757651] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2381.422767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2381.660795] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2381.715904] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2381.778660] Lustre: lustre-MDT0001: new disk, initializing [ 2381.858913] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2381.918760] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2381.925337] Lustre: 93548:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2381.930223] LustreError: 93548:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2385.855522] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2395.929681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2396.250499] Lustre: lustre-OST0000: new disk, initializing [ 2396.255301] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2396.263078] Lustre: 94313:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2396.324020] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2402.525757] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2405.921112] Lustre: 94770:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000a492d722 x1873261886480000/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786481976 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2411.006272] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2417.139926] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2417.151915] Lustre: MGS: Client 811915c5-42ae-426f-99ed-5f2c6a4dfadf (at 0@lo) reconnecting [ 2417.190803] Lustre: 92816:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2417.213858] LustreError: 92816:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 2417.224598] LustreError: 92816:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2417.245259] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2417.337615] Lustre: lustre-MDT0000: new disk, initializing [ 2417.453914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2417.497598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2417.605964] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2418.528959] Lustre: 95383:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000b4ee77a5 x1873261886486400/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1786481988 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2420.974624] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2426.337329] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2427.625473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2427.641368] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2427.772552] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2448.865580] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2448.877327] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2448.891832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2448.895499] Lustre: Skipped 1 previous similar message [ 2452.962589] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2452.963605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2452.981220] Lustre: Skipped 1 previous similar message [ 2454.871283] Lustre: server umount lustre-OST0000 complete [ 2458.093376] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2458.093635] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2458.100858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2464.062229] Lustre: server umount lustre-MDT0000 complete [ 2482.144697] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2482.453714] Lustre: server umount lustre-MDT0001 complete [ 2486.137344] Lustre: server umount MGS complete [ 2496.528177] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 17:00:49 (1786482049) [ 2511.639037] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 2513.841753] Key type lgssc unregistered [ 2514.087917] LNet: 97901:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2514.103686] LNetError: 97901:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2514.129799] LNet: Removed LNI 192.168.203.122@tcp [ 2514.950779] Key type .llcrypt unregistered [ 2514.953201] Key type ._llcrypt unregistered [ 2528.281614] Key type ._llcrypt registered [ 2528.285414] Key type .llcrypt registered [ 2528.401613] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 2529.474607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2529.575532] alg: No test for adler32 (adler32-zlib) [ 2530.690329] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 2530.909634] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 2532.568303] Key type lgssc registered [ 2533.664193] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2539.756553] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2539.767035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2541.011980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2544.461933] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2551.267995] LustreError: 99101:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2551.284951] LustreError: 99101:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2551.447536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2551.816837] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2555.600716] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2561.780310] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2566.774702] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2572.159182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2572.416929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2576.982967] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2582.624528] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2585.713365] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2596.326478] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2596.327439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2596.337956] Lustre: Skipped 1 previous similar message [ 2596.346754] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2600.814429] Lustre: server umount lustre-OST0000 complete [ 2606.562139] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2606.576394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2606.583473] Lustre: Skipped 2 previous similar messages [ 2609.543270] Lustre: server umount lustre-MDT0000 complete [ 2611.682295] LustreError: 100154:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2611.708444] LustreError: 100154:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2612.394360] LustreError: 99082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786482166 with bad export cookie 6636686385043257895 [ 2612.404341] LustreError: 99082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2612.406421] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2612.708570] Lustre: server umount lustre-MDT0001 complete [ 2620.294715] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 2622.369843] Key type lgssc unregistered [ 2622.566616] LNet: 102655:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2622.578188] LNetError: 102655:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2622.594927] LNet: Removed LNI 192.168.203.122@tcp [ 2623.207755] Key type .llcrypt unregistered [ 2623.210177] Key type ._llcrypt unregistered [ 2635.443416] Key type ._llcrypt registered [ 2635.446230] Key type .llcrypt registered [ 2635.530354] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 2636.434604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2636.487520] alg: No test for adler32 (adler32-zlib) [ 2637.591181] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 2637.802279] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 2639.512221] Key type lgssc registered [ 2640.375834] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2645.423293] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2645.430901] LustreError: 103702:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2645.497217] LustreError: 103717:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2645.505585] LustreError: 103702:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4932 [ 2645.513311] LustreError: 103702:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2645.523385] LustreError: 103702:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2645.542291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2646.864048] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2653.043854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2653.227713] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2656.484452] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2660.057795] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing lsmod [ 2662.230836] LustreError: 104909:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2662.308216] LustreError: 104921:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2662.312989] LustreError: 104909:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4922 [ 2662.321941] LustreError: 104909:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2662.332027] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2662.332777] LustreError: 104909:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2662.634281] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2664.612312] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2664.617526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2664.625869] Lustre: Skipped 1 previous similar message [ 2664.633243] Lustre: Skipped 2 previous similar messages [ 2670.054749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2670.061343] Lustre: Skipped 2 previous similar messages [ 2670.850830] Lustre: server umount lustre-MDT0000 complete [ 2673.494685] LustreError: 103726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786482227 with bad export cookie 9427493363782882739 [ 2673.500416] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2673.508417] LustreError: 103726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2673.820729] Lustre: server umount lustre-MDT0001 complete [ 2686.979386] Lustre: server umount lustre-OST0000 complete [ 2693.254972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2693.531681] LustreError: 105986:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2693.552039] LustreError: 105986:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2693.588943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2696.654939] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2698.722154] LustreError: 105987:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2702.819663] LustreError: 105986:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2702.889584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2703.139645] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2706.427619] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2711.654669] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2715.995510] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2720.690396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2720.957653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2725.481661] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2731.031057] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2738.355171] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 2748.898815] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2748.906776] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2748.910726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2748.913027] Lustre: Skipped 2 previous similar messages [ 2751.271651] Lustre: server umount lustre-OST0000 complete [ 2754.019100] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2754.021995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2754.031093] Lustre: Skipped 1 previous similar message [ 2754.037673] Lustre: Skipped 1 previous similar message [ 2759.140981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2759.145088] Lustre: Skipped 2 previous similar messages [ 2768.352212] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2768.531795] Lustre: server umount lustre-MDT0000 complete [ 2769.378379] LustreError: 105981:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2769.386742] LustreError: 105981:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2771.209776] LustreError: 106719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786482325 with bad export cookie 9427493363782883425 [ 2771.223114] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2771.228022] LustreError: 106719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2771.428664] Lustre: server umount lustre-MDT0001 complete [ 2778.676584] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 2780.853755] Key type lgssc unregistered [ 2781.073837] LNet: 109546:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2781.078652] LNetError: 109546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2781.092387] LNet: Removed LNI 192.168.203.122@tcp [ 2781.719535] Key type .llcrypt unregistered [ 2781.721706] Key type ._llcrypt unregistered [ 2800.456962] Key type ._llcrypt registered [ 2800.459269] Key type .llcrypt registered [ 2800.535445] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 2801.160775] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2801.174612] alg: No test for adler32 (adler32-zlib) [ 2802.168258] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 2802.368323] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 2804.048165] Key type lgssc registered [ 2805.027561] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2811.005209] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2811.109511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2812.363849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2815.267911] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2821.604597] LustreError: 110942:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2821.622912] LustreError: 110942:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2821.663270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2821.889108] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2825.416296] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2831.124270] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2835.449906] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2840.309669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2840.567111] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2844.973964] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2850.030891] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2856.941126] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 2864.877548] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2867.261695] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2872.034827] Lustre: server umount lustre-OST0000 complete [ 2877.409942] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2877.425142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2877.431783] Lustre: Skipped 1 previous similar message [ 2882.531834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2882.537189] Lustre: Skipped 1 previous similar message [ 2883.597455] Lustre: server umount lustre-MDT0000 complete [ 2886.191295] LustreError: 110922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786482440 with bad export cookie 5482209097917286997 [ 2886.200038] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2886.202137] LustreError: 110922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2896.439679] Lustre: server umount lustre-MDT0001 complete [ 2903.451380] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 2905.364629] Key type lgssc unregistered [ 2905.563134] LNet: 114601:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2905.568917] LNetError: 114601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2905.589610] LNet: Removed LNI 192.168.203.122@tcp [ 2906.195889] Key type .llcrypt unregistered [ 2906.201520] Key type ._llcrypt unregistered [ 2919.207860] Key type ._llcrypt registered [ 2919.210704] Key type .llcrypt registered [ 2919.299149] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 2920.142518] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2920.155122] alg: No test for adler32 (adler32-zlib) [ 2921.135709] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 2921.326216] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 2922.960259] Key type lgssc registered [ 2923.855406] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2928.883571] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2928.896799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2930.162930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2933.306359] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2939.094263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2939.320124] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2942.405492] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2947.577253] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2951.631980] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2956.452362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2956.654276] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2961.433854] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2967.092297] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2975.090095] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 2984.417231] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2984.421426] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2984.429908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2985.444605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2985.445209] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2985.445294] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2985.449454] Lustre: Skipped 1 previous similar message [ 2989.687940] Lustre: server umount lustre-OST0000 complete [ 2995.683393] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2995.684798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2995.689191] Lustre: Skipped 1 previous similar message [ 2995.689729] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2995.700098] Lustre: Skipped 1 previous similar message [ 2998.707123] Lustre: server umount lustre-MDT0000 complete [ 3000.804787] LustreError: 116603:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3000.820562] LustreError: 116603:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3001.415141] LustreError: 116536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786482555 with bad export cookie 15403697304453433226 [ 3001.421716] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3001.427193] LustreError: 116536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3001.653771] Lustre: server umount lustre-MDT0001 complete [ 3009.001215] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 3010.906950] Key type lgssc unregistered [ 3011.117770] LNet: 119356:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3011.124458] LNetError: 119356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3011.136709] LNet: Removed LNI 192.168.203.122@tcp [ 3011.722718] Key type .llcrypt unregistered [ 3011.725536] Key type ._llcrypt unregistered [ 3027.321359] Key type ._llcrypt registered [ 3027.322473] Key type .llcrypt registered [ 3027.376277] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 3027.829925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3027.840699] alg: No test for adler32 (adler32-zlib) [ 3028.763083] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 3028.885698] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 3030.496257] Key type lgssc registered [ 3031.031593] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3035.534208] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3035.550415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3036.796426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3039.063318] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3044.012918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3044.222620] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3046.561689] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3050.549634] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3053.375110] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3057.040200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3057.241272] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3060.873369] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3062.377222] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3065.067824] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3073.366727] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3075.408750] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3075.412783] Lustre: Skipped 2 previous similar messages [ 3078.006787] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3078.010936] Lustre: Skipped 2 previous similar messages [ 3079.225798] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3079.252486] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3083.145366] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.22@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3083.153610] Lustre: Skipped 2 previous similar messages [ 3087.187743] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.22@tcp 192.168.0.[2,10]@tcp [ 3090.332803] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.22/24@tcp in log lustre-MDT0000 [ 3090.339380] Lustre: Skipped 2 previous similar messages [ 3092.546391] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.22/24@tcp [ 3092.549015] Lustre: Skipped 1 previous similar message [ 3098.593896] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3098.602333] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3098.615486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3100.642960] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3100.643596] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3100.647403] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3102.772697] Lustre: server umount lustre-OST0000 complete [ 3105.763572] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3105.763626] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3105.774014] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3105.777170] Lustre: Skipped 1 previous similar message [ 3105.786385] Lustre: Skipped 2 previous similar messages [ 3110.883255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3110.890033] Lustre: Skipped 1 previous similar message [ 3111.172721] Lustre: server umount lustre-MDT0000 complete [ 3113.221808] LustreError: 123381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786482667 with bad export cookie 9717365576709561933 [ 3113.226784] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3113.232277] LustreError: 123381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3113.400852] Lustre: server umount lustre-MDT0001 complete [ 3119.893662] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 3121.763663] Key type lgssc unregistered [ 3121.952983] LNet: 125591:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3121.957294] LNetError: 125591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3121.966935] LNet: Removed LNI 192.168.203.122@tcp [ 3122.469721] Key type .llcrypt unregistered [ 3122.471646] Key type ._llcrypt unregistered [ 3139.415524] Key type ._llcrypt registered [ 3139.417515] Key type .llcrypt registered [ 3139.563915] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 3140.399567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3140.416473] alg: No test for adler32 (adler32-zlib) [ 3141.390992] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 3141.541505] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 3143.176202] Key type lgssc registered [ 3143.914039] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3148.750753] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3148.766359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3149.972558] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3149.983023] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3149.987278] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.22/24@tcp [ 3152.494672] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3157.537153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3157.717805] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3157.739627] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.22/24@tcp [ 3157.795336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3160.383495] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3164.642665] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3167.900292] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3172.083916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3172.297558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3175.408330] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3175.712631] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3179.911625] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3190.753205] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3190.758626] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3190.769035] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3192.802674] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3192.803400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3192.810635] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3192.818403] Lustre: Skipped 1 previous similar message [ 3196.155640] Lustre: server umount lustre-OST0000 complete [ 3198.945470] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3198.953641] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3198.966492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3203.042065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3203.043487] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3204.341630] Lustre: server umount lustre-MDT0000 complete [ 3206.443466] LustreError: 127111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786482760 with bad export cookie 16300486373417643762 [ 3206.445984] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3206.452026] LustreError: 127111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3206.620188] Lustre: server umount lustre-MDT0001 complete [ 3212.477112] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 3214.066844] Key type lgssc unregistered [ 3214.263346] LNet: 130715:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3214.272754] LNetError: 130715:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3214.286428] LNet: Removed LNI 192.168.203.122@tcp [ 3214.802514] Key type .llcrypt unregistered [ 3214.807770] Key type ._llcrypt unregistered [ 3228.356986] Key type ._llcrypt registered [ 3228.358969] Key type .llcrypt registered [ 3228.441498] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 3236.328575] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 3236.887751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3236.901888] alg: No test for adler32 (adler32-zlib) [ 3237.803974] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 3237.920982] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 3239.544323] Key type lgssc registered [ 3240.166131] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3258.278330] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3258.288856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3259.427623] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3259.446348] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3259.498662] Lustre: lustre-MDT0000: new disk, initializing [ 3259.536942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3259.547484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3261.638507] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3267.592502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3267.647375] Lustre: 134091:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3267.690764] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3267.694424] Lustre: Skipped 1 previous similar message [ 3267.743983] Lustre: lustre-MDT0001: new disk, initializing [ 3267.798739] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3267.822104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3267.829220] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3270.081547] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3275.517085] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3278.344420] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3281.909556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3282.047661] Lustre: lustre-OST0000: new disk, initializing [ 3282.050101] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3282.055517] Lustre: 136057:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3282.115768] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3285.186459] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3287.535930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3287.542833] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3287.576672] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3290.544555] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3292.798730] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3292.905023] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3294.801562] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3294.900076] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3297.763495] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3297.764592] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3297.766815] LustreError: Skipped 1 previous similar message [ 3297.779050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3297.783261] Lustre: Skipped 1 previous similar message [ 3302.883116] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3302.886671] Lustre: Skipped 1 previous similar message [ 3308.004499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3308.010037] Lustre: Skipped 1 previous similar message [ 3310.049033] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3310.126185] Lustre: server umount lustre-OST0000 complete [ 3314.145405] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3314.149292] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3314.153560] Lustre: Skipped 1 previous similar message [ 3314.155929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3316.194744] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3318.267071] Lustre: server umount lustre-MDT0000 complete [ 3320.151799] LustreError: 134081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786482874 with bad export cookie 14614455615994639917 [ 3320.156585] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3320.158198] LustreError: 134081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3320.341512] Lustre: server umount lustre-MDT0001 complete [ 3324.602440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3324.866703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3326.700074] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3330.021555] LustreError: 138272:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3330.037367] LustreError: 138272:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3331.020753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3331.212632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3333.473940] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3337.436648] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3340.412704] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3344.003434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3344.177647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3347.416055] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3350.882329] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3352.878553] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3352.970941] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3354.787726] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3356.943378] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 2 sec [ 3362.571503] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3362.669181] Lustre: lustre-OST0001: new disk, initializing [ 3362.673625] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3362.678937] Lustre: 141657:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3362.736022] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3363.613608] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3363.622328] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3363.670941] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3365.671392] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3370.288751] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3370.407798] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3372.463283] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3372.587291] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3375.537250] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff97ed07786000.ost_server_uuid 50 [ 3376.308301] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff97ed07786000.ost_server_uuid in FULL state after 0 sec [ 3384.294748] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3384.296424] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3384.301175] Lustre: Skipped 1 previous similar message [ 3384.303830] Lustre: Skipped 2 previous similar messages [ 3384.309413] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3385.965302] Lustre: server umount lustre-OST0001 complete [ 3389.411683] LustreError: 140089:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3389.413969] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3389.420809] LustreError: 140089:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3394.146351] Lustre: server umount lustre-OST0000 complete [ 3398.113369] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3398.117178] LustreError: Skipped 1 previous similar message [ 3398.119683] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3398.128191] Lustre: Skipped 1 previous similar message [ 3398.131316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3398.135303] Lustre: Skipped 3 previous similar messages [ 3410.400136] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3410.473698] Lustre: server umount lustre-MDT0000 complete [ 3412.392133] LustreError: 138252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786482966 with bad export cookie 14614455615994640799 [ 3412.397941] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3412.400082] LustreError: 138252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3412.567159] Lustre: server umount lustre-MDT0001 complete [ 3421.674167] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 3423.074452] Key type lgssc unregistered [ 3423.238616] LNet: 144657:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3423.243639] LNetError: 144657:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3423.255611] LNet: Removed LNI 192.168.203.122@tcp [ 3423.692904] Key type .llcrypt unregistered [ 3423.694099] Key type ._llcrypt unregistered [ 3442.929653] Key type ._llcrypt registered [ 3442.930967] Key type .llcrypt registered [ 3442.983289] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 3450.854344] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 3451.448854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3451.465703] alg: No test for adler32 (adler32-zlib) [ 3452.379333] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 3452.506508] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 3454.112250] Key type lgssc registered [ 3454.637874] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3471.697561] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3471.710330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3472.828654] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3472.849437] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3472.891761] Lustre: lustre-MDT0000: new disk, initializing [ 3472.929143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3472.942922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3474.866309] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3480.612602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3480.686393] Lustre: 148783:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3480.723939] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3480.726295] Lustre: Skipped 1 previous similar message [ 3480.770208] Lustre: lustre-MDT0001: new disk, initializing [ 3480.822775] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3480.848163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3480.854252] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3482.786643] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3487.913707] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3490.495707] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3493.796950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3493.953943] Lustre: lustre-OST0000: new disk, initializing [ 3493.958554] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3493.962921] Lustre: 150753:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3494.017772] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3496.887102] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3501.065230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3501.071565] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3501.126448] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3501.622608] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3504.144435] Lustre: Failing over lustre-OST0000 [ 3504.244929] Lustre: server umount lustre-OST0000 complete [ 3518.634227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3518.794283] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3518.804252] Lustre: Skipped 1 previous similar message [ 3518.814597] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3518.828587] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3521.585129] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3521.872109] Lustre: 147015:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786483060/real 1786483060] req@0000000046a51335 x1873263204527872/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786483076 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3523.340503] Lustre: Failing over lustre-MDT0000 [ 3523.369023] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3523.369830] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.22@tcp (stopping) [ 3523.473215] Lustre: server umount lustre-MDT0000 complete [ 3524.067767] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3524.075481] Lustre: Skipped 2 previous similar messages [ 3527.136157] Lustre: 147013:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786483065/real 1786483065] req@00000000bba46479 x1873263204529408/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786483081 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3527.150058] Lustre: 147013:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3529.185731] LustreError: 150016:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3529.198369] LustreError: 150016:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 3532.256579] Lustre: 147012:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786483070/real 1786483070] req@000000000141e631 x1873263204530432/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786483086 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3533.584550] LustreError: 148791:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3537.393465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3537.464351] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3537.469378] Lustre: 148787:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.122@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3537.573115] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 3537.608502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3537.640931] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3538.705731] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3539.626968] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3543.017648] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3543.030886] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 3543.062213] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3548.131508] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3548.131727] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3548.133953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3548.136550] LustreError: Skipped 1 previous similar message [ 3548.141625] Lustre: Skipped 1 previous similar message [ 3550.557399] Lustre: server umount lustre-OST0000 complete [ 3553.250262] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3553.250509] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3553.255192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3553.260553] Lustre: Skipped 1 previous similar message [ 3553.264804] Lustre: Skipped 2 previous similar messages [ 3558.372277] LustreError: 148796:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3558.390061] LustreError: 148796:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3558.440551] Lustre: server umount lustre-MDT0000 complete [ 3560.240847] LustreError: 149580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786483114 with bad export cookie 15557653649083579811 [ 3560.242595] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3560.246166] LustreError: 149580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3560.386662] Lustre: server umount lustre-MDT0001 complete [ 3565.294762] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 3566.656960] Key type lgssc unregistered [ 3566.813711] LNet: 154397:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3566.817815] LNetError: 154397:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3566.833747] LNet: Removed LNI 192.168.203.122@tcp [ 3567.259244] Key type .llcrypt unregistered [ 3567.261380] Key type ._llcrypt unregistered [ 3590.916082] Key type ._llcrypt registered [ 3590.918335] Key type .llcrypt registered [ 3590.981500] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 3591.492664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3591.504165] alg: No test for adler32 (adler32-zlib) [ 3592.411437] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 3592.547438] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 3594.160112] Key type lgssc registered [ 3594.703282] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3598.299447] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3598.309577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3599.556514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3601.617630] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3606.383475] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3608.670473] LustreError: 155944:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3608.675994] LustreError: 155944:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3608.680816] LustreError: 155944:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 3608.820252] Lustre: server umount lustre-MDT0000 complete [ 3615.168359] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 3619.434467] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 3635.448360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3635.548644] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3635.563369] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3635.602327] Lustre: lustre-MDT0000: new disk, initializing [ 3635.633221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3635.641634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3637.328581] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3642.019791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3642.067405] Lustre: 159242:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3642.095953] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3642.098052] Lustre: Skipped 1 previous similar message [ 3642.133819] Lustre: lustre-MDT0001: new disk, initializing [ 3642.172273] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3642.186586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3642.191281] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3643.850482] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3648.123550] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3650.489385] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3653.373813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3653.514723] Lustre: lustre-OST0000: new disk, initializing [ 3653.517667] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3653.522638] Lustre: 161208:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3653.560673] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3655.952414] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3658.758857] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3658.762977] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3658.813887] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3660.299883] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3662.438845] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 3663.174034] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 17:20:17 (1786483217) [ 3760.096744] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3760.100827] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3760.108790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3761.120858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3761.120905] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3761.128835] Lustre: Skipped 1 previous similar message [ 3762.921378] Lustre: server umount lustre-MDT0000 complete [ 3764.464546] LustreError: 159233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786483318 with bad export cookie 13483702654438095460 [ 3764.465483] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3764.469735] LustreError: 159233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3766.241396] LustreError: 161216:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3766.241510] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3766.249748] LustreError: 161216:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3766.257206] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3766.263730] Lustre: Skipped 2 previous similar messages [ 3769.248809] LustreError: 161763:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3769.248867] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3770.631955] Lustre: server umount lustre-MDT0001 complete [ 3778.402530] Lustre: server umount lustre-OST0000 complete [ 3782.254570] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 3786.323394] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 3801.230841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3801.343178] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3801.358164] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3801.399626] Lustre: lustre-MDT0000: new disk, initializing [ 3801.436417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3801.447687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3803.075314] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3807.819966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3807.870141] Lustre: 165844:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3807.896128] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3807.898446] Lustre: Skipped 1 previous similar message [ 3807.941461] Lustre: lustre-MDT0001: new disk, initializing [ 3808.000301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3808.005585] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3809.606775] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3814.082273] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3816.441855] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3819.155209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3819.275293] Lustre: lustre-OST0000: new disk, initializing [ 3819.277291] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3819.280822] Lustre: 167809:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3819.319427] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3819.322469] Lustre: Skipped 1 previous similar message [ 3821.790802] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3826.502590] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3827.727168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3827.732754] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3827.783159] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3828.339419] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3828.435667] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3830.157149] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3830.249757] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3832.806059] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3832.806191] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3832.812698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3832.815333] LustreError: Skipped 1 previous similar message [ 3837.092401] Lustre: server umount lustre-OST0000 complete [ 3838.944951] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3838.947724] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3838.953109] Lustre: Skipped 1 previous similar message [ 3843.041362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3843.045315] Lustre: Skipped 3 previous similar messages [ 3844.831923] Lustre: server umount lustre-MDT0000 complete [ 3846.224380] LustreError: 165836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786483400 with bad export cookie 13483702654438254262 [ 3846.227937] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3846.229260] LustreError: 165836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3846.359222] Lustre: server umount lustre-MDT0001 complete [ 3849.654174] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 17:23:23 (1786483403) [ 3855.706163] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 3859.420437] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 3873.668205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3873.755305] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3873.771116] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3873.812347] Lustre: lustre-MDT0000: new disk, initializing [ 3873.835623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3873.842826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3875.282414] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3879.622805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3879.671363] Lustre: 172559:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3879.782564] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3881.215057] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3885.154319] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3887.154263] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3889.564538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3889.675355] Lustre: 174534:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3890.071527] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3890.116586] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3891.925069] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3895.955288] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3900.385901] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3900.385935] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3900.388441] LustreError: Skipped 1 previous similar message [ 3900.392400] Lustre: Skipped 1 previous similar message [ 3900.394633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3900.397529] Lustre: Skipped 1 previous similar message [ 3903.958200] Lustre: server umount lustre-OST0000 complete [ 3905.506259] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3911.387521] Lustre: server umount lustre-MDT0000 complete [ 3912.659198] LustreError: 172549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786483467 with bad export cookie 13483702654438255116 [ 3912.662168] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3912.663836] LustreError: 172549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3912.778257] Lustre: server umount lustre-MDT0001 complete [ 3915.662608] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 17:24:29 (1786483469) [ 3921.510124] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 3925.296309] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 3938.666078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3938.777749] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3938.793588] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3938.796221] Lustre: Skipped 3 previous similar messages [ 3938.835412] Lustre: lustre-MDT0000: new disk, initializing [ 3938.837311] Lustre: Skipped 2 previous similar messages [ 3938.868263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3938.871304] Lustre: Skipped 2 previous similar messages [ 3938.878557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3938.881732] Lustre: Skipped 2 previous similar messages [ 3940.379114] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3944.711831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3944.766462] Lustre: 178843:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3944.880621] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3946.461469] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3950.467444] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3952.630809] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3955.165803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3955.274085] Lustre: 180822:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3957.545651] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3961.645485] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3961.899899] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3972.067136] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3972.067952] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3972.073675] Lustre: Skipped 2 previous similar messages [ 3972.074495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3972.080983] Lustre: Skipped 5 previous similar messages [ 3975.590331] Lustre: server umount lustre-OST0000 complete [ 3983.202115] Lustre: server umount lustre-MDT0000 complete [ 3984.529333] LustreError: 180821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786483538 with bad export cookie 13483702654438256187 [ 3984.532301] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3984.533746] LustreError: 180821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3984.660622] Lustre: server umount lustre-MDT0001 complete [ 3988.335965] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 3989.337653] Key type lgssc unregistered [ 3989.468377] LNet: 182678:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3989.471574] LNetError: 182678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3989.478280] LNet: Removed LNI 192.168.203.122@tcp [ 3989.765098] Key type .llcrypt unregistered [ 3989.766215] Key type ._llcrypt unregistered [ 3998.877562] Key type ._llcrypt registered [ 3998.878738] Key type .llcrypt registered [ 3998.918159] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 3999.300609] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3999.356643] alg: No test for adler32 (adler32-zlib) [ 4000.204488] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 4000.287898] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 4001.864148] Key type lgssc registered [ 4002.227156] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4004.751439] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4004.758502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4005.913512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4007.409608] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4010.630718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4010.763633] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4012.196419] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4015.053571] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4017.196794] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4019.821819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4019.958525] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4022.180639] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4024.867545] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4031.207931] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4051.938879] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4051.940939] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4051.941930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4051.942822] LustreError: Skipped 1 previous similar message [ 4051.947623] Lustre: Skipped 1 previous similar message [ 4053.268369] Lustre: server umount lustre-OST0000 complete [ 4057.057781] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4057.058243] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4057.066917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4057.069511] Lustre: Skipped 2 previous similar messages [ 4060.761222] Lustre: server umount lustre-MDT0000 complete [ 4062.148485] LustreError: 184095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786483616 with bad export cookie 14514793523464411251 [ 4062.152163] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4062.153231] LustreError: 184095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4062.275731] Lustre: server umount lustre-MDT0001 complete [ 4066.644532] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 4067.788858] Key type lgssc unregistered [ 4067.930536] LNet: 187635:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4067.934105] LNetError: 187635:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4067.946165] LNet: Removed LNI 192.168.203.122@tcp [ 4068.287090] Key type .llcrypt unregistered [ 4068.288483] Key type ._llcrypt unregistered [ 4077.443213] Key type ._llcrypt registered [ 4077.444333] Key type .llcrypt registered [ 4077.477632] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 4077.909160] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4077.924093] alg: No test for adler32 (adler32-zlib) [ 4078.779245] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 4078.865588] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 4080.440182] Key type lgssc registered [ 4080.795389] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4083.491658] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4083.498297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4084.637499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4085.986116] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4088.871039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4089.002039] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4090.484605] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4093.239370] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4095.295358] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4097.799077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4097.919619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4099.947361] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4102.357449] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4110.249383] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4115.428064] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4115.428856] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4115.435495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4115.439396] Lustre: Skipped 1 previous similar message [ 4117.787875] Lustre: lustre-OST0000: Not available for connect from 192.168.203.22@tcp (stopping) [ 4120.545189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4120.547863] Lustre: Skipped 1 previous similar message [ 4122.896812] Lustre: lustre-OST0000: Not available for connect from 192.168.203.22@tcp (stopping) [ 4128.017359] Lustre: lustre-OST0000: Not available for connect from 192.168.203.22@tcp (stopping) [ 4128.020509] Lustre: Skipped 2 previous similar messages [ 4129.248105] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4129.311864] Lustre: server umount lustre-OST0000 complete [ 4131.867438] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff97ed18dbe800.ost_server_uuid 50 [ 4132.425485] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff97ed18dbe800.ost_server_uuid in DISCONN state after 0 sec [ 4133.047613] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 4143.148369] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4143.149408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4143.152964] Lustre: Skipped 2 previous similar messages [ 4149.280291] Lustre: server umount lustre-MDT0000 complete [ 4150.543496] LustreError: 190873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786483704 with bad export cookie 16391814805848124052 [ 4150.547985] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4150.550726] LustreError: 190873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4150.671516] Lustre: server umount lustre-MDT0001 complete [ 4153.654501] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 17:28:27 (1786483707) [ 4155.979045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4156.145545] LustreError: 192992:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4156.152036] LustreError: 192992:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4156.170870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4157.554609] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4160.600293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4160.755122] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4162.277478] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4165.009619] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4167.051980] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4169.487684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4169.624349] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4171.692130] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4174.305737] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4176.713593] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4176.799420] Lustre: lustre-OST0001: new disk, initializing [ 4176.802473] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4176.806744] Lustre: 195832:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4176.810904] Lustre: lustre-OST0001: Not available for connect from 0@lo (not set up) [ 4176.813486] Lustre: Skipped 3 previous similar messages [ 4178.903122] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4180.970062] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4180.970663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4180.978918] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4180.998666] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4182.883382] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4186.084818] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4186.085695] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4189.958098] Lustre: server umount lustre-OST0000 complete [ 4191.203197] LustreError: 194812:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4191.209922] LustreError: 194812:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4192.150237] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4192.233189] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4193.554354] LustreError: 194809:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4193.778820] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4193.856768] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4196.321725] LustreError: 195838:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4196.327449] LustreError: 195838:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4201.442388] LustreError: 194811:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4201.448354] LustreError: 194811:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4206.562489] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4206.564414] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4206.565754] LustreError: Skipped 2 previous similar messages [ 4206.570842] Lustre: Skipped 1 previous similar message [ 4210.005076] Lustre: server umount lustre-OST0001 complete [ 4211.681854] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4211.682544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4211.685641] Lustre: Skipped 1 previous similar message [ 4211.687423] Lustre: Skipped 5 previous similar messages [ 4217.432037] Lustre: server umount lustre-MDT0000 complete [ 4218.746194] LustreError: 194833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786483773 with bad export cookie 16391814805848125109 [ 4218.749094] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4218.750503] LustreError: 194833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4218.872515] Lustre: server umount lustre-MDT0001 complete [ 4227.531161] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 17:29:41 (1786483781) [ 4230.000300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4230.114309] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4230.129584] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4230.217377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4230.220315] Lustre: Skipped 1 previous similar message [ 4231.731669] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4235.162766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4235.218605] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4235.240537] Lustre: 199242:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4236.967653] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4239.779842] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4242.013251] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4244.534327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4244.610128] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4246.881230] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4248.345862] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 4249.543988] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4251.927359] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4251.957427] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4254.066799] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4256.753433] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4258.290203] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4258.295380] Lustre: Skipped 1 previous similar message [ 4258.320956] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 4263.804558] Lustre: server umount lustre-OST0000 complete [ 4267.280663] LustreError: 202877:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4267.287076] LustreError: 202877:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4278.754681] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4278.755843] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4278.757630] LustreError: Skipped 1 previous similar message [ 4278.758094] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4278.762915] Lustre: Skipped 1 previous similar message [ 4278.764322] Lustre: Skipped 8 previous similar messages [ 4280.602172] Lustre: server umount lustre-OST0001 complete [ 4288.098927] Lustre: server umount lustre-MDT0000 complete [ 4288.994237] LustreError: 199253:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4289.000346] LustreError: 199253:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 4289.481226] LustreError: 199990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786483843 with bad export cookie 16391814805848126495 [ 4289.484981] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4289.485781] LustreError: 199990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4289.600088] Lustre: server umount lustre-MDT0001 complete [ 4297.860221] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 17:30:52 (1786483852) [ 4303.413406] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 4306.919020] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 4319.422868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4319.518321] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4319.533203] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4319.567470] Lustre: lustre-MDT0000: new disk, initializing [ 4319.588343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4319.590718] Lustre: Skipped 3 previous similar messages [ 4319.596122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4321.032777] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4325.316630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4325.366156] Lustre: 207299:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4325.397075] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4325.399428] Lustre: Skipped 1 previous similar message [ 4325.440234] Lustre: lustre-MDT0001: new disk, initializing [ 4325.492875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4325.496696] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4327.070242] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4331.137931] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4333.110548] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4335.416269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4335.538386] Lustre: lustre-OST0000: new disk, initializing [ 4335.540466] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4335.543703] Lustre: 209271:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4337.606832] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4338.112964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4338.117816] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4338.163578] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4341.399323] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4342.968977] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4343.059121] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4344.516290] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4344.593248] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4348.387153] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4348.387202] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4348.390640] LustreError: Skipped 1 previous similar message [ 4348.394538] Lustre: Skipped 2 previous similar messages [ 4351.381317] Lustre: server umount lustre-OST0000 complete [ 4358.753654] Lustre: server umount lustre-MDT0000 complete [ 4359.986160] LustreError: 207293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786483914 with bad export cookie 16391814805848128238 [ 4359.990878] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4360.117665] Lustre: server umount lustre-MDT0001 complete [ 4363.319624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4363.488050] LustreError: 211482:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4364.945355] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4367.883624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4369.435945] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4371.733417] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4373.461394] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4375.549025] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4377.424259] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4379.682318] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4381.085743] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4388.296265] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec [ 4389.855960] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4389.950362] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4393.442439] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4396.909766] Lustre: server umount lustre-OST0000 complete [ 4400.635618] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4403.560526] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4427.906758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4428.027311] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4428.029675] Lustre: Skipped 5 previous similar messages [ 4430.412212] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4433.230325] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4433.851520] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4434.148549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4434.155113] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4434.155236] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4439.525661] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4439.526083] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4439.531220] Lustre: Skipped 5 previous similar messages [ 4439.533920] LustreError: Skipped 1 previous similar message [ 4439.539044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4439.540869] Lustre: Skipped 12 previous similar messages [ 4441.111602] Lustre: server umount lustre-OST0000 complete [ 4448.728976] Lustre: server umount lustre-MDT0000 complete [ 4449.761202] LustreError: 212209:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4449.767870] LustreError: 212209:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4450.174585] LustreError: 211463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786484004 with bad export cookie 16391814805848129106 [ 4450.177995] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4450.179210] LustreError: 211463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4450.316515] Lustre: server umount lustre-MDT0001 complete [ 4453.732290] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 17:33:27 (1786484007) [ 4456.430130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4458.423218] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4462.113810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4464.350735] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4470.639645] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4476.167293] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4482.944266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4490.706493] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4499.799684] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4504.600745] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4504.746358] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4508.981678] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4509.297544] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4516.319872] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4516.467608] Lustre: lustre-OST0001: new disk, initializing [ 4516.470588] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4516.478617] Lustre: 220861:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4522.555133] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4524.229284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4524.240836] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4524.410507] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4532.348729] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4537.794471] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4537.943807] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4542.954635] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4543.383947] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4550.117513] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4550.129078] LustreError: Skipped 1 previous similar message [ 4560.352160] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4560.483753] Lustre: server umount lustre-OST0001 complete [ 4567.009921] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4567.194638] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4573.493406] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4573.666907] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4579.620383] LustreError: 221487:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4579.634179] LustreError: 221487:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 4602.400592] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4602.553597] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4602.571235] Lustre: Skipped 4 previous similar messages [ 4607.984423] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4608.007593] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4608.021479] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4608.028832] Lustre: Skipped 1 previous similar message [ 4609.604081] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4617.495184] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4619.714848] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4623.329600] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4623.330373] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4623.339139] LustreError: Skipped 2 previous similar messages [ 4623.376673] Lustre: Skipped 6 previous similar messages [ 4628.324816] Lustre: server umount lustre-OST0001 complete [ 4651.347594] Lustre: server umount lustre-OST0000 complete [ 4661.859648] Lustre: server umount lustre-MDT0000 complete [ 4665.282120] LustreError: 217474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786484219 with bad export cookie 16391814805848130051 [ 4665.292784] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4665.321570] LustreError: 217474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4671.689317] Lustre: server umount lustre-MDT0001 complete [ 4694.621821] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 17:37:28 (1786484248) [ 4701.110355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4701.361075] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4701.382509] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4706.536728] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4715.736587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4715.821648] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4715.891674] Lustre: 226108:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4722.235469] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4729.969529] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4737.437650] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4747.708610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4747.992547] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4754.282385] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4763.046347] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4774.632681] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4774.740498] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4781.090636] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4788.096765] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4790.001617] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4793.988854] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4794.179616] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4798.719251] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4798.890573] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4804.420219] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff97ed08e4a800.ost_server_uuid 50 [ 4805.940315] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff97ed08e4a800.ost_server_uuid in IDLE state after 0 sec [ 4807.379636] Lustre: Permanently deactivating lustre-OST0001 [ 4807.398334] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 4811.316852] Lustre: Permanently reactivating lustre-OST0001 [ 4816.869618] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4816.873906] LustreError: Skipped 3 previous similar messages [ 4816.880643] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4816.884969] Lustre: Skipped 17 previous similar messages [ 4822.619577] Lustre: server umount lustre-OST0001 complete [ 4834.233565] Lustre: server umount lustre-OST0000 complete [ 4843.708239] Lustre: server umount lustre-MDT0000 complete [ 4844.513483] LustreError: 226938:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4844.531580] LustreError: 226938:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 31 previous similar messages [ 4848.181843] LustreError: 226100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786484402 with bad export cookie 16391814805848131325 [ 4848.197562] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4848.258431] LustreError: 226100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4848.719779] Lustre: server umount lustre-MDT0001 complete [ 4870.566175] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 17:40:24 (1786484424) [ 4877.054221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4877.246678] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4877.276286] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4877.286618] Lustre: Skipped 5 previous similar messages [ 4877.444826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4877.454160] Lustre: Skipped 4 previous similar messages [ 4881.747937] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4889.599385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4889.685285] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4889.703537] Lustre: 232805:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4893.969671] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4900.883242] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4905.986374] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4913.575298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4913.750628] Lustre: Permanently deactivating lustre-OST0000 [ 4913.763887] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 4919.895669] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4924.032090] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4926.940476] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4932.949612] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4932.992198] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4932.995157] Lustre: Skipped 1 previous similar message [ 4939.568286] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4941.541266] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 4946.722411] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4948.916912] Lustre: Permanently reactivating lustre-OST0000 [ 4948.931933] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 4948.935883] Lustre: Skipped 2 previous similar messages [ 4951.088694] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 4951.122725] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4951.131498] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4951.140445] Lustre: Skipped 1 previous similar message [ 4951.164291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4951.184399] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4962.275436] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4962.285197] Lustre: Skipped 10 previous similar messages [ 4962.659537] Lustre: server umount lustre-OST0001 complete [ 4972.398945] Lustre: server umount lustre-OST0000 complete [ 4982.316976] Lustre: server umount lustre-MDT0000 complete [ 4985.287272] LustreError: 232795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786484539 with bad export cookie 16391814805848132963 [ 4985.301131] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4985.302289] LustreError: 232795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4985.566899] Lustre: server umount lustre-MDT0001 complete [ 4995.528932] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 4997.841218] Key type lgssc unregistered [ 4998.171765] LNet: 237664:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4998.178693] LNetError: 237664:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4999.206682] LNet: Removed LNI 192.168.203.122@tcp [ 5000.021456] Key type .llcrypt unregistered [ 5000.023776] Key type ._llcrypt unregistered [ 5019.128151] Key type ._llcrypt registered [ 5019.130337] Key type .llcrypt registered [ 5019.220599] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 5020.100325] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5020.124984] alg: No test for adler32 (adler32-zlib) [ 5021.099533] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 5021.326471] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 5022.984182] Key type lgssc registered [ 5023.968887] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5031.133647] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5031.153591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5032.421667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5034.975910] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5040.115812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5040.200728] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5040.225278] Lustre: 239104:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5040.232843] Lustre: Permanently deactivating lustre-MDT0001 [ 5040.244990] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5040.441971] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5043.336430] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5045.249154] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5047.483558] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5050.490244] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5054.269620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5054.483845] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5057.572863] LustreError: 240954:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5057.582445] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 5057.586187] LustreError: 240954:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5058.076050] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5062.155889] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5062.626369] LustreError: 240954:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5064.677206] LustreError: 240954:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5064.681470] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 5064.686300] LustreError: 240954:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5065.573033] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5065.712277] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5068.447159] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5070.838901] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 5070.845209] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 5072.039326] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5073.972796] Lustre: Permanently reactivating lustre-MDT0001 [ 5073.982825] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 5073.986990] Lustre: Skipped 2 previous similar messages [ 5074.907318] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5074.915251] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 5074.920090] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 5074.930450] LustreError: 242677:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 5074.933201] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5074.939620] LustreError: 242677:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 5074.955355] LustreError: 242677:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.122@tcp: cfg command failed: rc = -108 [ 5074.959688] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 5074.959688] [ 5074.964554] LustreError: 239106:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 5080.858765] Lustre: lustre-MDT0001: Client 727f5ac9-4338-4fc5-97a9-a909187c1b6a (at 192.168.203.22@tcp) reconnecting [ 5083.942158] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5084.718591] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5087.384417] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 5087.477373] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 5089.354487] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 5089.455019] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 5090.609765] Lustre: Permanently deactivating lustre-MDT0001 [ 5090.617068] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5090.619718] Lustre: Skipped 2 previous similar messages [ 5097.991673] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5106.661603] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5106.663928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5106.666566] Lustre: Skipped 3 previous similar messages [ 5106.671374] Lustre: Skipped 3 previous similar messages [ 5110.452829] Lustre: server umount lustre-MDT0000 complete [ 5111.778660] LustreError: 239115:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5111.785143] LustreError: 239115:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5112.104744] LustreError: 239095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786484666 with bad export cookie 17385557810200289807 [ 5112.110530] LustreError: 239095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5112.111488] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5112.271781] Lustre: server umount lustre-MDT0001 complete [ 5119.991918] Lustre: server umount lustre-OST0000 complete [ 5124.899899] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 17:44:39 (1786484679) [ 5129.208626] Lustre: server umount lustre-OST0001 complete [ 5131.844802] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 5135.571281] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 5149.096066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5149.196479] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5149.201905] Lustre: Skipped 2 previous similar messages [ 5149.212098] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5149.260188] Lustre: lustre-MDT0000: new disk, initializing [ 5149.289662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5149.300086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5150.912776] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5155.654414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5155.697636] Lustre: 247407:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5155.723169] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5155.725286] Lustre: Skipped 1 previous similar message [ 5155.743079] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5155.767491] Lustre: lustre-MDT0001: new disk, initializing [ 5155.815385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5155.820065] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5157.256857] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5161.202788] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5163.231605] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5165.841359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5165.973775] Lustre: lustre-OST0000: new disk, initializing [ 5165.976424] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5165.980053] Lustre: 249373:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5166.017651] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5166.021769] Lustre: Skipped 1 previous similar message [ 5168.255588] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5170.689559] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5170.695084] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5170.730606] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5172.097620] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5172.762359] LustreError: 247412:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5177.152351] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5177.233258] Lustre: lustre-OST0001: new disk, initializing [ 5177.235635] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5177.239380] Lustre: 250489:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5179.502432] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5180.610527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5180.615089] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5180.650209] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5182.856132] LustreError: 247412:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5182.866104] LustreError: 247442:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5183.439844] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5184.002806] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5192.960134] LustreError: 247442:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5196.263553] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5196.264087] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5196.272661] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5196.274835] Lustre: Skipped 1 previous similar message [ 5198.617841] Lustre: lustre-OST0001: Not available for connect from 192.168.203.22@tcp (stopping) [ 5199.749421] Lustre: server umount lustre-OST0001 complete [ 5201.380318] LustreError: 249917:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5201.386570] LustreError: 249917:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5211.620222] LustreError: 249358:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5211.625086] LustreError: 249358:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5216.739642] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5216.740101] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5216.741814] LustreError: Skipped 1 previous similar message [ 5216.746374] Lustre: Skipped 1 previous similar message [ 5216.749796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5216.751574] Lustre: Skipped 1 previous similar message [ 5217.874660] Lustre: server umount lustre-OST0000 complete [ 5221.857775] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5221.862692] Lustre: Skipped 2 previous similar messages [ 5225.371455] Lustre: server umount lustre-MDT0000 complete [ 5226.592276] LustreError: 248202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786484781 with bad export cookie 17385557810200292621 [ 5226.595465] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5226.595555] LustreError: 248202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5232.748636] Lustre: server umount lustre-MDT0001 complete [ 5236.280121] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 5237.367290] Key type lgssc unregistered [ 5237.504462] LNet: 252523:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5237.507344] LNetError: 252523:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5237.516430] LNet: Removed LNI 192.168.203.122@tcp [ 5237.813174] Key type .llcrypt unregistered [ 5237.814173] Key type ._llcrypt unregistered [ 5251.581503] Key type ._llcrypt registered [ 5251.582498] Key type .llcrypt registered [ 5251.618282] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 5252.089035] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5252.106129] alg: No test for adler32 (adler32-zlib) [ 5252.992939] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 5253.106107] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 5254.688146] Key type lgssc registered [ 5255.059858] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5257.690371] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5257.697167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5258.791199] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5258.799922] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5258.858431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5260.221660] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5263.024061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5263.058133] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5263.068056] Lustre: 254707:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5263.173597] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5264.555883] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5266.883920] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5268.614694] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5270.770690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5270.832796] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5270.894075] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5272.793532] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5275.078590] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5277.193174] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 5323.233707] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5323.233960] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5323.236644] LustreError: Skipped 1 previous similar message [ 5323.237495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5323.240923] Lustre: Skipped 1 previous similar message [ 5324.635313] Lustre: server umount lustre-OST0000 complete [ 5326.088046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 5330.220312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5330.340286] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5332.094864] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5334.300687] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5342.563967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5342.567327] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5342.567469] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5342.571776] Lustre: Skipped 1 previous similar message [ 5368.289642] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5368.292196] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5368.298442] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5368.300655] Lustre: Skipped 2 previous similar messages [ 5373.266627] Lustre: server umount lustre-OST0000 complete [ 5375.968836] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5375.971545] LustreError: Skipped 1 previous similar message [ 5375.972970] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5375.976931] Lustre: Skipped 1 previous similar message [ 5375.978827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5378.529257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5378.529781] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5380.571799] Lustre: server umount lustre-MDT0000 complete [ 5381.877539] LustreError: 254697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786484936 with bad export cookie 1896169237295056913 [ 5381.881681] LustreError: 254697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5381.881950] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5382.016529] Lustre: server umount lustre-MDT0001 complete [ 5385.842204] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 5386.859972] Key type lgssc unregistered [ 5386.983407] LNet: 260128:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5386.986253] LNetError: 260128:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5386.994348] LNet: Removed LNI 192.168.203.122@tcp [ 5387.326570] Key type .llcrypt unregistered [ 5387.328345] Key type ._llcrypt unregistered [ 5396.467972] Key type ._llcrypt registered [ 5396.469579] Key type .llcrypt registered [ 5396.508908] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 5396.967836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5396.980528] alg: No test for adler32 (adler32-zlib) [ 5397.843841] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 5397.936877] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 5399.520139] Key type lgssc registered [ 5399.917545] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5402.514703] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5402.520035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5403.674854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5405.059597] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5407.803594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5407.913457] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5409.312757] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5411.645246] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5413.383147] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5415.542729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5415.646440] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5417.580231] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5420.022823] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5428.964620] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 5438.432943] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5438.436469] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5438.442375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5439.460313] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5439.462061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5439.462317] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5439.468763] Lustre: Skipped 1 previous similar message [ 5444.045857] Lustre: server umount lustre-OST0000 complete [ 5449.184914] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5449.186872] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5449.190746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5451.411912] Lustre: server umount lustre-MDT0000 complete [ 5452.592880] LustreError: 261501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786485007 with bad export cookie 8406864277867000433 [ 5452.595676] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5452.596245] LustreError: 261501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5452.705815] Lustre: server umount lustre-MDT0001 complete [ 5456.159562] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 5457.188686] Key type lgssc unregistered [ 5457.322412] LNet: 265844:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5457.324932] LNetError: 265844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5457.334330] LNet: Removed LNI 192.168.203.122@tcp [ 5457.644425] Key type .llcrypt unregistered [ 5457.645689] Key type ._llcrypt unregistered [ 5464.318198] Key type ._llcrypt registered [ 5464.319313] Key type .llcrypt registered [ 5464.355964] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 5464.716703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5464.735349] alg: No test for adler32 (adler32-zlib) [ 5465.585849] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 5465.682125] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 5467.272188] Key type lgssc registered [ 5467.606319] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5470.018278] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5470.024032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5471.152994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5472.358227] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5474.992099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5475.106524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5476.483299] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5478.814984] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5480.544352] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5482.690143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5482.816581] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5484.653039] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5486.944044] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5496.166667] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5501.411110] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5501.411453] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5501.412438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5501.421529] Lustre: Skipped 1 previous similar message [ 5503.060295] Lustre: server umount lustre-OST0000 complete [ 5506.017097] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5506.019661] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5506.024685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5506.027033] Lustre: Skipped 1 previous similar message [ 5506.388409] Lustre: server umount lustre-MDT0000 complete [ 5506.529854] LustreError: 267037:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5506.535438] LustreError: 267037:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5507.581368] LustreError: 267024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786485062 with bad export cookie 15359194156731080751 [ 5507.584734] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5507.585841] LustreError: 267024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5507.704980] Lustre: server umount lustre-MDT0001 complete [ 5511.171873] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 5512.147563] Key type lgssc unregistered [ 5512.278400] LNet: 270738:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5512.280835] LNetError: 270738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5512.288286] LNet: Removed LNI 192.168.203.122@tcp [ 5512.584715] Key type .llcrypt unregistered [ 5512.586065] Key type ._llcrypt unregistered [ 5521.096600] Key type ._llcrypt registered [ 5521.098212] Key type .llcrypt registered [ 5521.136259] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 5521.448331] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5521.475075] alg: No test for adler32 (adler32-zlib) [ 5522.317626] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 5522.399302] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 5523.976188] Key type lgssc registered [ 5524.323817] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5526.721135] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5526.726694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5527.862665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5529.101377] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5531.713697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5531.823550] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5532.988091] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5535.262653] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5537.053867] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5539.239414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5539.351401] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5541.216079] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5543.525950] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5552.675832] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5562.849552] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5562.852638] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5562.854477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5562.857510] Lustre: Skipped 1 previous similar message [ 5567.057267] Lustre: server umount lustre-OST0000 complete [ 5573.089766] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5573.089794] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5573.090581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5573.090586] Lustre: Skipped 1 previous similar message [ 5573.094093] Lustre: Skipped 1 previous similar message [ 5573.096777] LustreError: Skipped 1 previous similar message [ 5574.355511] Lustre: server umount lustre-MDT0000 complete [ 5575.481041] LustreError: 272111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786485129 with bad export cookie 13946407615037902100 [ 5575.483359] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5575.484465] LustreError: 272111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5575.590267] Lustre: server umount lustre-MDT0001 complete [ 5578.968628] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 5579.937272] Key type lgssc unregistered [ 5580.050325] LNet: 276456:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5580.053259] LNetError: 276456:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5580.060305] LNet: Removed LNI 192.168.203.122@tcp [ 5580.329735] Key type .llcrypt unregistered [ 5580.331039] Key type ._llcrypt unregistered [ 5586.951194] Key type ._llcrypt registered [ 5586.952302] Key type .llcrypt registered [ 5586.988554] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 5587.363801] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5587.370691] alg: No test for adler32 (adler32-zlib) [ 5588.214174] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 5588.300100] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 5589.880122] Key type lgssc registered [ 5590.223971] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5592.703418] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5592.710907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5593.877783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5595.230912] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5598.000570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5598.110127] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5599.382959] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5601.670320] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5603.399545] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5605.577556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5605.700432] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5607.491722] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5609.811152] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5619.046359] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 5624.291585] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5624.291801] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5624.292401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5624.295991] Lustre: Skipped 1 previous similar message [ 5625.937141] Lustre: server umount lustre-OST0000 complete [ 5628.896787] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5628.898973] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5628.902555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5628.904216] Lustre: Skipped 1 previous similar message [ 5629.207956] Lustre: server umount lustre-MDT0000 complete [ 5629.409266] LustreError: 277678:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5629.417536] LustreError: 277678:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5630.417117] LustreError: 277635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786485184 with bad export cookie 6339303153833395070 [ 5630.419850] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5630.421451] LustreError: 277635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5630.538594] Lustre: server umount lustre-MDT0001 complete [ 5633.829499] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 5634.835754] Key type lgssc unregistered [ 5634.964385] LNet: 281388:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5634.967269] LNetError: 281388:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5634.975259] LNet: Removed LNI 192.168.203.122@tcp [ 5635.270513] Key type .llcrypt unregistered [ 5635.271831] Key type ._llcrypt unregistered [ 5638.629618] Key type ._llcrypt registered [ 5638.630526] Key type .llcrypt registered [ 5638.668436] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 5643.771629] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 5649.429355] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 5649.664714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5649.671200] alg: No test for adler32 (adler32-zlib) [ 5650.515170] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 5650.598977] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 5652.176111] Key type lgssc registered [ 5652.492649] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5662.691628] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5662.697416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5663.776206] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5663.786598] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5663.823271] Lustre: lustre-MDT0000: new disk, initializing [ 5663.842671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5663.847773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5665.098789] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5668.966793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5669.008536] Lustre: 284914:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5669.029201] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5669.032236] Lustre: Skipped 1 previous similar message [ 5669.064045] Lustre: lustre-MDT0001: new disk, initializing [ 5669.091492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5669.101206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5669.104276] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5670.285113] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5673.716290] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5675.432516] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5677.598478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5677.693054] Lustre: lustre-OST0000: new disk, initializing [ 5677.694704] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5677.697214] Lustre: 286879:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5677.722338] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5677.841401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5677.844451] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5677.871266] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5679.501432] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5682.973602] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5684.260496] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5684.321736] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5685.517429] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5685.576321] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5688.289440] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5688.291089] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5688.291627] LustreError: Skipped 1 previous similar message [ 5688.295498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5688.298712] Lustre: Skipped 1 previous similar message [ 5692.240547] Lustre: server umount lustre-OST0000 complete [ 5693.410264] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5693.410710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5693.414369] Lustre: Skipped 2 previous similar messages [ 5693.417729] Lustre: Skipped 1 previous similar message [ 5698.529424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5698.531323] Lustre: Skipped 1 previous similar message [ 5699.540867] Lustre: server umount lustre-MDT0000 complete [ 5700.715134] LustreError: 285708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786485255 with bad export cookie 1594201068712622021 [ 5700.717791] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5700.718160] LustreError: 285708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5700.824525] Lustre: server umount lustre-MDT0001 complete [ 5703.328899] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 17:54:17 (1786485257) [ 5705.321115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5705.478096] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5706.679608] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5709.292525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5710.596119] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5712.724918] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5714.358973] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5716.376460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5716.467938] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5716.471293] Lustre: Skipped 1 previous similar message [ 5718.244054] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5720.386542] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5734.882552] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5734.883590] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5734.887849] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5734.889781] LustreError: Skipped 1 previous similar message [ 5734.892972] Lustre: Skipped 1 previous similar message [ 5739.984534] Lustre: server umount lustre-OST0000 complete [ 5744.098057] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 5744.100793] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5744.104991] Lustre: Skipped 1 previous similar message [ 5744.106876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5747.538270] Lustre: server umount lustre-MDT0000 complete [ 5748.759276] LustreError: 289268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786485303 with bad export cookie 1594201068712622889 [ 5748.762359] LustreError: 290022:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 5748.762650] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5748.763506] LustreError: 289268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5748.880577] Lustre: server umount lustre-MDT0001 complete [ 5752.670925] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 5753.681445] Key type lgssc unregistered [ 5753.806320] LNet: 292840:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5753.808758] LNetError: 292840:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5753.818313] LNet: Removed LNI 192.168.203.122@tcp [ 5754.124205] Key type .llcrypt unregistered [ 5754.125516] Key type ._llcrypt unregistered [ 5762.606231] Key type ._llcrypt registered [ 5762.607192] Key type .llcrypt registered [ 5762.640710] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 5768.623627] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 5768.993774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5769.037623] alg: No test for adler32 (adler32-zlib) [ 5769.892598] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 5769.992450] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 5771.568097] Key type lgssc registered [ 5771.951357] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5786.419477] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5786.426697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5787.501940] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5787.512078] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5787.548486] Lustre: lustre-MDT0000: new disk, initializing [ 5787.567719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5787.573680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5788.821455] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5792.633678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5792.671132] Lustre: 296637:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5792.694905] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5792.697632] Lustre: Skipped 1 previous similar message [ 5792.731182] Lustre: lustre-MDT0001: new disk, initializing [ 5792.761100] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5792.772480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5792.777548] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5794.055719] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5797.592257] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5799.254813] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5801.323078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5801.425298] Lustre: lustre-OST2710: new disk, initializing [ 5801.427390] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 5801.431336] Lustre: 298601:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5801.459069] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 5803.430362] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5803.518987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 5803.522991] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 5803.554758] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 5807.037480] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5809.142425] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5809.200041] Lustre: lustre-OST03e8: new disk, initializing [ 5809.201767] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 5809.204213] Lustre: 299664:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5809.241093] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 5811.080819] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5811.970989] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 5811.975658] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 5812.003969] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 5813.520551] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5816.124404] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 5816.193044] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5817.560384] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 5817.640181] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5819.094235] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 5819.166486] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5820.511532] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 5820.581730] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5822.439156] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5822.440565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5822.442762] Lustre: Skipped 3 previous similar messages [ 5822.447688] Lustre: Skipped 3 previous similar messages [ 5827.553214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5827.555323] Lustre: Skipped 1 previous similar message [ 5827.738875] Lustre: server umount lustre-MDT0000 complete [ 5828.960297] LustreError: 299663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786485383 with bad export cookie 1288196682135893060 [ 5828.963733] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5828.964611] LustreError: 299663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5829.079547] Lustre: server umount lustre-MDT0001 complete [ 5839.880335] Lustre: server umount lustre-OST2710 complete [ 5851.360937] Lustre: server umount lustre-OST03e8 complete [ 5857.358235] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 5860.703271] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 5876.914373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5877.099277] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5877.115735] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5877.165557] Lustre: lustre-MDT0000: new disk, initializing [ 5877.209269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5877.222931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5879.348511] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5885.770032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5885.887412] Lustre: 304902:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5885.947912] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5885.950607] Lustre: Skipped 1 previous similar message [ 5886.003384] Lustre: lustre-MDT0001: new disk, initializing [ 5886.093351] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5886.120916] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5886.134883] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5889.191522] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5895.391837] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5898.861565] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5903.087871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5903.265777] Lustre: lustre-OST0000: new disk, initializing [ 5903.268762] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5903.273699] Lustre: 306869:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5903.336382] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5903.647524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5903.653196] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5903.731058] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5906.648178] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5912.093851] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5914.512618] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5914.634985] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5917.018424] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5917.148633] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5919.203111] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5919.205069] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5919.208685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5919.214057] Lustre: Skipped 1 previous similar message [ 5919.218275] Lustre: Skipped 3 previous similar messages [ 5924.261556] Lustre: server umount lustre-OST0000 complete [ 5927.393745] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5927.399066] LustreError: Skipped 1 previous similar message [ 5927.400938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5927.407599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5929.442648] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5932.326110] Lustre: server umount lustre-MDT0000 complete [ 5933.748305] LustreError: 304894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786485488 with bad export cookie 1288196682135894719 [ 5933.751366] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5933.752979] LustreError: 304894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5933.881541] Lustre: server umount lustre-MDT0001 complete [ 5937.578979] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 17:58:11 (1786485491) [ 5938.237993] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 5938.944636] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 17:58:13 (1786485493) [ 5940.219907] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 5951.623927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5951.745195] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5951.758621] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5951.830793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5953.571376] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5957.438433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5957.495987] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5957.517963] Lustre: 310721:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5959.467477] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5976.418980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5976.498428] Lustre: Denying initial registration attempt from nid 192.168.203.122@tcp, specified as failover [ 5976.503830] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 5976.508311] LustreError: 312173:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 5976.511572] LustreError: 312173:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 5976.515444] LustreError: 312173:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 5976.542403] Lustre: server umount lustre-OST0000 complete [ 5976.544794] LustreError: 312173:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 5983.201626] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5983.208781] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5983.217477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5983.221043] Lustre: Skipped 2 previous similar messages [ 5985.592519] Lustre: server umount lustre-MDT0000 complete [ 5987.222444] LustreError: 310714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786485541 with bad export cookie 1288196682135895608 [ 5987.224855] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5987.227609] LustreError: 310714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5987.356945] Lustre: server umount lustre-MDT0001 complete [ 5992.196994] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 5993.414812] Key type lgssc unregistered [ 5993.550562] LNet: 313298:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5993.554151] LNetError: 313298:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5993.564895] LNet: Removed LNI 192.168.203.122@tcp [ 5993.952436] Key type .llcrypt unregistered [ 5993.954034] Key type ._llcrypt unregistered [ 5999.144309] Key type ._llcrypt registered [ 5999.146015] Key type .llcrypt registered [ 5999.196346] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 5999.644689] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5999.680044] alg: No test for adler32 (adler32-zlib) [ 6000.550543] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 6000.651978] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 6002.248153] Key type lgssc registered [ 6002.736415] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6029.687791] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 6033.117314] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6033.128751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6034.241666] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6034.251753] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6034.322525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6036.021420] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6039.621864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6039.670532] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6039.685104] Lustre: 316035:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6039.817671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6041.568231] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6044.570889] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6046.792351] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6062.565223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6062.649064] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6062.732524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6065.229507] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6068.580321] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6069.583824] Lustre: server umount lustre-OST0000 complete [ 6072.827937] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6072.836876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6072.839670] Lustre: Skipped 1 previous similar message [ 6077.548524] Lustre: server umount lustre-MDT0000 complete [ 6079.287892] LustreError: 317902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786485633 with bad export cookie 147671460784659840 [ 6079.288433] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6079.293163] LustreError: 317902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6079.462516] Lustre: server umount lustre-MDT0001 complete [ 6084.380863] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 6085.836780] Key type lgssc unregistered [ 6085.998647] LNet: 319602:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6086.002113] LNetError: 319602:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6086.013816] LNet: Removed LNI 192.168.203.122@tcp [ 6086.434867] Key type .llcrypt unregistered [ 6086.437557] Key type ._llcrypt unregistered [ 6097.314765] Key type ._llcrypt registered [ 6097.315992] Key type .llcrypt registered [ 6097.377573] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 6097.777193] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6097.813133] alg: No test for adler32 (adler32-zlib) [ 6098.694487] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 6098.803460] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 6100.400155] Key type lgssc registered [ 6100.838454] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6104.013892] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6104.023046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6105.193432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6106.743681] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6110.046033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6110.201615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6111.805923] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6114.659146] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6116.819193] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6121.445359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6121.588566] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6123.944956] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6126.769037] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6140.897393] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6140.902364] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6140.910425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6143.263418] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.22@tcp (stopping) [ 6145.506691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6145.508849] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6145.511565] Lustre: Skipped 1 previous similar message [ 6145.519131] Lustre: Skipped 1 previous similar message [ 6146.999520] Lustre: server umount lustre-MDT0000 complete [ 6148.367944] LustreError: 320992:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6148.377265] LustreError: 320992:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6148.640583] LustreError: 320976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786485703 with bad export cookie 11797484686449523912 [ 6148.646778] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6148.649883] LustreError: 320976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6150.625128] LustreError: 320991:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6150.625441] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6150.641514] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6150.645912] Lustre: Skipped 1 previous similar message [ 6153.488154] LustreError: 320993:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6154.828092] Lustre: server umount lustre-MDT0001 complete [ 6157.632700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6161.383579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6174.176854] LustreError: 320307:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000002ece67bd x1873265980105344/t0(0) o250->MGC192.168.203.122@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6174.310290] LustreError: 322835:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6174.319989] LustreError: 322835:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6174.342224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6174.367045] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 6176.171168] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6179.489873] LustreError: 324617:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6179.498851] LustreError: 324617:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6180.004067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6180.168163] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6181.879725] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6184.295758] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6185.196476] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6189.977440] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 6192.231273] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6194.964358] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 6199.779763] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6199.781265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6199.785802] Lustre: Skipped 1 previous similar message [ 6199.790179] Lustre: Skipped 3 previous similar messages [ 6202.788566] Lustre: server umount lustre-OST0000 complete [ 6204.898369] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6204.905495] Lustre: Skipped 1 previous similar message [ 6210.018505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6210.023485] Lustre: Skipped 3 previous similar messages [ 6210.608481] Lustre: server umount lustre-MDT0000 complete [ 6212.304046] LustreError: 323809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786485766 with bad export cookie 11797484686449531066 [ 6212.309601] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6212.310821] LustreError: 323809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6212.447412] Lustre: server umount lustre-MDT0001 complete [ 6217.199881] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 6218.405586] Key type lgssc unregistered [ 6218.561804] LNet: 327288:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6218.567373] LNetError: 327288:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6218.580627] LNet: Removed LNI 192.168.203.122@tcp [ 6218.972896] Key type .llcrypt unregistered [ 6218.973996] Key type ._llcrypt unregistered [ 6230.763901] Key type ._llcrypt registered [ 6230.765786] Key type .llcrypt registered [ 6230.813630] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 6231.167602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6231.176147] alg: No test for adler32 (adler32-zlib) [ 6232.057477] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 6232.169619] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 6233.760121] Key type lgssc registered [ 6234.236472] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6237.379032] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6237.387059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6238.560614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6240.248345] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6243.656820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6243.805946] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6245.364425] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6248.135526] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6250.206416] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6253.026067] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6253.026688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6253.031461] Lustre: Skipped 1 previous similar message [ 6253.036159] Lustre: Skipped 1 previous similar message [ 6254.050218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6254.052956] Lustre: Skipped 1 previous similar message [ 6257.438295] Lustre: server umount lustre-MDT0000 complete [ 6258.787230] LustreError: 328661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786485813 with bad export cookie 9487505499369491844 [ 6258.791049] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6258.792195] LustreError: 328661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6258.919968] Lustre: server umount lustre-MDT0001 complete [ 6264.259718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6264.373918] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6264.385731] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6264.465249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6265.931840] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6269.031593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6269.081779] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6269.096642] Lustre: 331267:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6269.207593] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6270.759758] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6273.526133] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6275.643938] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6280.677464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6280.753280] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 6280.758022] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 6280.761840] LustreError: 333147:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 6280.765286] LustreError: 333147:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 6280.768659] LustreError: 333147:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 6280.798813] Lustre: server umount lustre-OST0000 complete [ 6280.801019] LustreError: 333147:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 6285.795595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6285.865817] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6285.937511] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6288.183097] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6291.161133] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6291.995976] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 6294.175424] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6294.212879] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6294.246374] Lustre: lustre-OST0001: new disk, initializing [ 6294.248415] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6294.251915] Lustre: 334687:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6294.287378] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6296.480923] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6297.294561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6297.300202] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6297.334284] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6299.198839] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6302.693440] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6302.693564] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6302.694730] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6302.698402] Lustre: Skipped 1 previous similar message [ 6302.702199] LustreError: Skipped 1 previous similar message [ 6306.370619] Lustre: server umount lustre-OST0001 complete [ 6307.810722] LustreError: 333615:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6307.816488] LustreError: 333615:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6312.417192] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6312.417447] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6312.420461] LustreError: Skipped 1 previous similar message [ 6312.421143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6312.421147] Lustre: Skipped 1 previous similar message [ 6312.433041] Lustre: Skipped 1 previous similar message [ 6312.932545] LustreError: 334356:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6312.939684] LustreError: 334356:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6318.051394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6318.052250] LustreError: 335693:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6318.054067] Lustre: Skipped 2 previous similar messages [ 6318.064151] LustreError: 335693:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6322.144131] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6322.206485] Lustre: server umount lustre-OST0000 complete [ 6325.730709] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6325.734508] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6328.289292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6328.292192] Lustre: Skipped 2 previous similar messages [ 6329.690365] Lustre: server umount lustre-MDT0000 complete [ 6331.129413] LustreError: 332004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786485885 with bad export cookie 9487505499369492383 [ 6331.133611] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6331.134944] LustreError: 332004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6331.258665] Lustre: server umount lustre-MDT0001 complete [ 6335.722450] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 6336.929803] Key type lgssc unregistered [ 6337.068402] LNet: 336649:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6337.072466] LNetError: 336649:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6337.082372] LNet: Removed LNI 192.168.203.122@tcp [ 6337.448300] Key type .llcrypt unregistered [ 6337.449965] Key type ._llcrypt unregistered [ 6370.493033] Key type ._llcrypt registered [ 6370.494102] Key type .llcrypt registered [ 6370.535017] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 6377.403386] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 6377.874896] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6377.884365] alg: No test for adler32 (adler32-zlib) [ 6378.747328] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 6378.840093] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 6380.432118] Key type lgssc registered [ 6380.875961] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6395.679454] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6395.690969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6396.803180] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6396.816880] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6396.860493] Lustre: lustre-MDT0000: new disk, initializing [ 6396.894418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6396.904169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6398.586730] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6403.478142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6403.528775] Lustre: 341551:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6403.566899] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6403.569589] Lustre: Skipped 1 previous similar message [ 6403.608618] Lustre: lustre-MDT0001: new disk, initializing [ 6403.646735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6403.661952] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6403.665554] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6405.167764] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6409.143378] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6411.145219] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6413.564750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6413.674873] Lustre: lustre-OST0000: new disk, initializing [ 6413.677142] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6413.680931] Lustre: 343520:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6413.715133] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6415.708359] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6419.713829] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6421.177812] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6421.497136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6421.500760] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6421.530962] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6422.274203] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 6423.690632] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6423.765106] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6426.594266] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6426.594730] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6426.596962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6426.598489] LustreError: Skipped 1 previous similar message [ 6426.603245] Lustre: Skipped 1 previous similar message [ 6430.549328] Lustre: server umount lustre-OST0000 complete [ 6434.273129] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6434.277216] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6434.284045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6434.287146] Lustre: Skipped 1 previous similar message [ 6436.833192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6436.833657] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6437.979734] Lustre: server umount lustre-MDT0000 complete [ 6439.323901] LustreError: 342345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786485993 with bad export cookie 5372303909935286097 [ 6439.327681] LustreError: 342345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6439.329203] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6439.440614] Lustre: server umount lustre-MDT0001 complete [ 6442.301161] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 18:06:36 (1786485996) [ 6444.886502] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 18:06:39 (1786485999) [ 6447.833174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6448.019111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6449.535716] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6452.651760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6452.803277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6454.289528] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6457.142345] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6459.161867] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6461.819297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6461.954041] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6464.220398] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6467.070105] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6481.116603] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 6481.749323] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 6482.375161] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 6483.037894] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 6483.771785] Lustre: Failing over lustre-MDT0000 [ 6483.946513] Lustre: server umount lustre-MDT0000 complete [ 6484.451197] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6484.453280] LustreError: 346303:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6484.457337] Lustre: Skipped 1 previous similar message [ 6484.467956] LustreError: 346303:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6487.832494] LustreError: 346303:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6489.569192] LustreError: 346277:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6489.574871] LustreError: 346277:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6492.944078] LustreError: 346272:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6497.324939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6497.394481] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6497.505611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6497.533967] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6498.064584] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6499.095563] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6502.887507] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6502.903273] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 6502.922484] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6504.906592] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6505.478571] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6507.567682] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 6508.329822] Lustre: Failing over lustre-MDT0000 [ 6508.505100] Lustre: server umount lustre-MDT0000 complete [ 6513.124612] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6513.126255] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6513.128985] Lustre: Skipped 1 previous similar message [ 6513.134513] LustreError: 346277:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6513.150955] LustreError: 346277:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6521.932296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6521.987745] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6522.117139] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6523.578357] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6523.666480] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6527.462191] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6527.464591] Lustre: Skipped 2 previous similar messages [ 6527.478446] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 6527.496450] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6529.709919] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6530.410459] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6537.698247] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6537.699487] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6537.699893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6537.699898] Lustre: Skipped 1 previous similar message [ 6537.711027] Lustre: Skipped 2 previous similar messages [ 6539.103725] Lustre: server umount lustre-MDT0000 complete [ 6540.459196] LustreError: 346304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786486094 with bad export cookie 5372303909935288792 [ 6540.463501] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6540.464131] LustreError: 346304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6540.594329] Lustre: server umount lustre-MDT0001 complete [ 6552.292703] Lustre: server umount lustre-OST0000 complete [ 6558.300544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6558.499964] LustreError: 352625:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6558.508625] LustreError: 352625:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6558.531554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6558.534099] Lustre: Skipped 1 previous similar message [ 6560.020646] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6563.102362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6564.692331] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6567.405330] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6569.448449] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6572.185091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6574.702126] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6577.578089] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6582.569241] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6587.874543] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6587.875711] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6587.878443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6587.881577] Lustre: Skipped 2 previous similar messages [ 6587.884670] Lustre: Skipped 3 previous similar messages [ 6590.361763] Lustre: server umount lustre-OST0000 complete [ 6598.113280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6598.116935] Lustre: Skipped 3 previous similar messages [ 6606.304093] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6606.361887] Lustre: server umount lustre-MDT0000 complete [ 6607.675349] LustreError: 352607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786486162 with bad export cookie 5372303909935289541 [ 6607.678244] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6607.680359] LustreError: 352607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6607.795304] Lustre: server umount lustre-MDT0001 complete [ 6611.785878] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 6612.944433] Key type lgssc unregistered [ 6613.099503] LNet: 356161:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6613.102837] LNetError: 356161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6613.111578] LNet: Removed LNI 192.168.203.122@tcp [ 6613.445843] Key type .llcrypt unregistered [ 6613.447377] Key type ._llcrypt unregistered [ 6624.244634] Key type ._llcrypt registered [ 6624.245758] Key type .llcrypt registered [ 6624.285390] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 6624.717127] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6624.726303] alg: No test for adler32 (adler32-zlib) [ 6625.590797] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 6625.696104] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 6627.288165] Key type lgssc registered [ 6627.723252] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6630.772137] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6630.783415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6631.967114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6633.523946] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6636.708478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6636.859102] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6638.298577] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6641.111379] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6643.412104] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6646.049957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6646.179804] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6648.319691] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6650.897885] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6657.446603] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6667.747217] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6667.747289] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6667.748379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6667.752487] Lustre: Skipped 2 previous similar messages [ 6672.667873] Lustre: server umount lustre-MDT0000 complete [ 6672.866907] LustreError: 357655:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6672.874575] LustreError: 357655:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6674.028558] LustreError: 357637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786486228 with bad export cookie 16850636192404452303 [ 6674.031162] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6674.033675] LustreError: 357637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6674.168433] Lustre: server umount lustre-MDT0001 complete [ 6685.859850] Lustre: server umount lustre-OST0000 complete [ 6690.611921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6690.769539] LustreError: 361383:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6690.792872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6692.171264] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6695.108520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6695.226949] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6696.568176] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6699.382887] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6701.513498] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6703.808200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6703.915345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6705.798212] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6708.224491] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6716.201372] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 6726.112862] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6726.115789] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6726.121376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6726.123358] Lustre: Skipped 2 previous similar messages [ 6731.732055] Lustre: server umount lustre-MDT0000 complete [ 6731.745991] LustreError: 362110:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6731.751465] LustreError: 362110:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6732.993261] LustreError: 362117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786486287 with bad export cookie 16850636192404453402 [ 6732.996532] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6732.998610] LustreError: 362117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6733.117980] Lustre: server umount lustre-MDT0001 complete [ 6743.644071] Lustre: server umount lustre-OST0000 complete [ 6748.838150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6748.985393] LustreError: 365308:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6749.009978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6750.417893] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6753.261308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6754.821932] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6757.455653] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6759.376300] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6761.690307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6763.872789] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6766.388813] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6775.142688] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 6778.337035] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6778.339529] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6778.343387] Lustre: Skipped 2 previous similar messages [ 6778.345482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6778.347242] Lustre: Skipped 2 previous similar messages [ 6780.386373] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6780.387926] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6780.388823] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6784.338677] Lustre: server umount lustre-OST0000 complete [ 6789.088864] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6789.092533] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6789.098169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6789.099870] Lustre: Skipped 1 previous similar message [ 6791.763812] Lustre: server umount lustre-MDT0000 complete [ 6793.065456] LustreError: 365291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786486347 with bad export cookie 16850636192404454410 [ 6793.068496] LustreError: 365291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6793.069461] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6793.187635] Lustre: server umount lustre-MDT0001 complete [ 6796.880924] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 6797.993235] Key type lgssc unregistered [ 6798.144547] LNet: 368861:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6798.148426] LNetError: 368861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6798.155540] LNet: Removed LNI 192.168.203.122@tcp [ 6798.497339] Key type .llcrypt unregistered [ 6798.498673] Key type ._llcrypt unregistered [ 6808.570850] Key type ._llcrypt registered [ 6808.572413] Key type .llcrypt registered [ 6808.612833] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 6809.010123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6809.045239] alg: No test for adler32 (adler32-zlib) [ 6809.911800] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 6810.010314] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 6811.600433] Key type lgssc registered [ 6811.965485] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6814.682859] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6814.693521] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6814.697749] LustreError: 370278:0:(osd_handler.c:8554:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 6814.708771] LustreError: 370278:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 6814.712104] LustreError: 370278:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 6814.715348] LustreError: 370278:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 6814.718703] LustreError: 370278:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6817.318637] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6817.321238] LustreError: 370675:0:(osd_handler.c:8554:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 6817.331036] LustreError: 370675:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 6817.334121] LustreError: 370675:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 6817.336889] LustreError: 370675:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 6817.339800] LustreError: 370675:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6823.851947] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 6824.882314] Key type lgssc unregistered [ 6825.025488] LNet: 371678:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6825.028178] LNetError: 371678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6825.038336] LNet: Removed LNI 192.168.203.122@tcp [ 6825.326269] Key type .llcrypt unregistered [ 6825.327559] Key type ._llcrypt unregistered [ 6831.715097] Key type ._llcrypt registered [ 6831.716321] Key type .llcrypt registered [ 6831.761755] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 6838.543067] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 6838.939420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6838.978170] alg: No test for adler32 (adler32-zlib) [ 6839.869428] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 6839.982583] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 6841.584181] Key type lgssc registered [ 6842.008816] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6854.649106] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6854.656451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6855.750217] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6855.764616] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6855.801848] Lustre: lustre-MDT0000: new disk, initializing [ 6855.835110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6855.844074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6857.506925] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6862.058448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6862.112700] Lustre: 374822:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6862.143612] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6862.147207] Lustre: Skipped 1 previous similar message [ 6862.185459] Lustre: lustre-MDT0001: new disk, initializing [ 6862.219972] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6862.232921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6862.238535] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6863.862329] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6867.980743] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6870.080996] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6872.674252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6872.778236] Lustre: lustre-OST0000: new disk, initializing [ 6872.780485] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6872.784162] Lustre: 376791:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6872.814040] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6874.937718] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6878.870646] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6879.226554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6879.230823] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6879.269897] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6880.432746] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6880.506386] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6882.026971] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6882.111994] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6884.323496] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6884.323607] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6884.327045] LustreError: Skipped 1 previous similar message [ 6884.333193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6884.337431] Lustre: Skipped 1 previous similar message [ 6888.920633] Lustre: server umount lustre-OST0000 complete [ 6893.025093] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6893.028309] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6893.033174] Lustre: Skipped 1 previous similar message [ 6893.035192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6894.561780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6894.562611] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6896.413680] Lustre: server umount lustre-MDT0000 complete [ 6897.734875] LustreError: 375616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786486452 with bad export cookie 6178003852895900282 [ 6897.738397] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6897.739807] LustreError: 375616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6897.853186] Lustre: server umount lustre-MDT0001 complete [ 6900.934464] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 6901.615338] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 18:14:15 (1786486455) [ 6904.294810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6904.529938] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6906.150434] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6909.534021] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6909.686239] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6911.253651] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6914.090946] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6916.282865] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6918.922361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6919.057700] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6921.181654] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6923.891196] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6926.401141] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6926.478707] Lustre: lustre-OST0001: new disk, initializing [ 6926.481209] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6926.484080] Lustre: 382084:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6928.513772] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6928.835726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6928.840656] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6928.874745] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6932.292146] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6935.012620] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6935.014031] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6935.017723] Lustre: Skipped 1 previous similar message [ 6935.020689] Lustre: Skipped 1 previous similar message [ 6940.132795] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6940.135360] Lustre: Skipped 2 previous similar messages [ 6940.221645] Lustre: server umount lustre-OST0001 complete [ 6945.251982] LustreError: 381070:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6945.257176] LustreError: 381070:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6950.371974] LustreError: 382889:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6950.376881] LustreError: 382889:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6955.491340] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6955.491694] LustreError: 381071:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6955.491938] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6955.492572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6955.505103] LustreError: 381071:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6958.350257] Lustre: server umount lustre-OST0000 complete [ 6965.794457] Lustre: server umount lustre-MDT0000 complete [ 6967.161376] LustreError: 379230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786486521 with bad export cookie 6178003852895901150 [ 6967.164216] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6967.166426] LustreError: 379230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6973.367130] Lustre: server umount lustre-MDT0001 complete [ 6977.277403] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 6978.313340] Key type lgssc unregistered [ 6978.444385] LNet: 384068:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6978.447713] LNetError: 384068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6978.455343] LNet: Removed LNI 192.168.203.122@tcp [ 6978.772517] Key type .llcrypt unregistered [ 6978.773696] Key type ._llcrypt unregistered [ 6996.328446] Key type ._llcrypt registered [ 6996.329462] Key type .llcrypt registered [ 6996.366271] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 6996.664949] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6996.671738] alg: No test for adler32 (adler32-zlib) [ 6997.523813] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 6997.609593] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 6999.192159] Key type lgssc registered [ 6999.537651] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7002.123789] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7002.131704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7003.228517] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7003.237905] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7003.298869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7004.679339] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7007.569267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7007.610411] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7007.622616] Lustre: 386854:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7007.725899] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7009.068634] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7011.466871] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7013.261422] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7017.953488] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7017.953880] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7017.954464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7017.962254] Lustre: Skipped 1 previous similar message [ 7023.074550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7023.077596] Lustre: Skipped 2 previous similar messages [ 7028.192123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7028.193291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7028.198207] Lustre: Skipped 1 previous similar message [ 7028.256960] Lustre: server umount lustre-MDT0000 complete [ 7029.528242] LustreError: 386845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786486583 with bad export cookie 12100682966081995806 [ 7029.531233] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7029.532709] LustreError: 386845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7029.646241] Lustre: server umount lustre-MDT0001 complete [ 7032.728819] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 18:16:26 (1786486586) [ 7034.893025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7035.065534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7036.418777] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7039.222087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7039.337842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7040.714409] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7043.074432] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7044.838665] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7047.016414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7047.087854] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7047.150599] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7048.962582] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7051.353672] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7055.000694] Lustre: Permanently deactivating lustre-OST0000 [ 7055.005920] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7057.136121] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7058.179337] LustreError: 392250:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 7058.183543] LustreError: 392250:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 7058.186433] LustreError: 392250:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7058.770766] LustreError: 392299:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 7058.773627] LustreError: 392299:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7065.482966] Lustre: server umount lustre-OST0000 complete [ 7066.340182] LustreError: 392549:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 7066.342596] LustreError: 392549:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7069.153282] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7069.153834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7069.157113] Lustre: Skipped 1 previous similar message [ 7074.273056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7074.275172] Lustre: Skipped 2 previous similar messages [ 7077.367515] Lustre: server umount lustre-MDT0000 complete [ 7078.552981] LustreError: 389345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786486633 with bad export cookie 12100682966081996373 [ 7078.555645] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7078.556603] LustreError: 389345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7089.174337] Lustre: server umount lustre-MDT0001 complete [ 7092.030780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7093.444207] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7094.359523] LustreError: 393706:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 7096.404145] Lustre: 394044:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.122@tcp [ 7097.103539] Lustre: server umount MGS complete [ 7101.168141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7101.330111] LustreError: 394766:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7101.337936] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7101.339878] LustreError: 394766:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7101.342483] Lustre: Skipped 1 previous similar message [ 7101.368496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7102.799880] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7105.608160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7105.706658] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7107.085561] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7109.446847] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7111.242301] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7111.860523] Lustre: Permanently reactivating lustre-OST0000 [ 7111.865874] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7111.867554] Lustre: Skipped 2 previous similar messages [ 7138.272133] LustreError: 396594:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 7141.711962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7141.822248] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7141.824670] Lustre: Skipped 1 previous similar message [ 7143.762143] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7146.329306] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7163.361973] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7163.363666] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7163.365550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7163.368989] Lustre: Skipped 1 previous similar message [ 7163.374296] Lustre: Skipped 1 previous similar message [ 7164.377817] Lustre: server umount lustre-OST0000 complete [ 7167.456913] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7167.460245] LustreError: Skipped 1 previous similar message [ 7167.462415] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7180.256121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7180.313847] Lustre: server umount lustre-MDT0000 complete [ 7181.450950] LustreError: 394746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786486735 with bad export cookie 12100682966081997395 [ 7181.454885] LustreError: 394746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7181.455451] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7181.562284] Lustre: server umount lustre-MDT0001 complete [ 7184.883908] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 7185.815542] Key type lgssc unregistered [ 7185.929366] LNet: 399488:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7185.931508] LNetError: 399488:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7185.937289] LNet: Removed LNI 192.168.203.122@tcp [ 7186.221429] Key type .llcrypt unregistered [ 7186.222447] Key type ._llcrypt unregistered [ 7191.716478] Key type ._llcrypt registered [ 7191.717543] Key type .llcrypt registered [ 7191.754198] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 7197.659446] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 7198.004963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7198.022325] alg: No test for adler32 (adler32-zlib) [ 7198.870441] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 7198.955080] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 7200.536096] Key type lgssc registered [ 7200.850824] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7211.285947] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7211.292049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7212.366763] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7212.375661] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7212.413127] Lustre: lustre-MDT0000: new disk, initializing [ 7212.434117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7212.439796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7213.716391] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7217.723078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7217.765085] Lustre: 402616:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7217.787594] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7217.789793] Lustre: Skipped 1 previous similar message [ 7217.827075] Lustre: lustre-MDT0001: new disk, initializing [ 7217.857203] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7217.868416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7217.872371] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7219.168229] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7222.901880] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7224.794310] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7227.146118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7227.248745] Lustre: lustre-OST0000: new disk, initializing [ 7227.250331] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7227.253209] Lustre: 404587:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7227.279475] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7229.177788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7229.181261] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7229.182127] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7229.218094] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7232.910279] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7234.320546] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7234.398241] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7235.866214] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7235.949725] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7239.649866] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7239.651738] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7239.653285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7239.657432] Lustre: Skipped 1 previous similar message [ 7239.661321] Lustre: Skipped 1 previous similar message [ 7242.641842] Lustre: server umount lustre-OST0000 complete [ 7244.770297] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7244.771373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7244.774715] Lustre: Skipped 1 previous similar message [ 7244.778622] Lustre: Skipped 1 previous similar message [ 7249.889769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7249.892270] Lustre: Skipped 1 previous similar message [ 7250.016699] Lustre: server umount lustre-MDT0000 complete [ 7251.351419] LustreError: 402609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786486805 with bad export cookie 2209327336057505114 [ 7251.355423] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7251.357374] LustreError: 402609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7251.512414] Lustre: server umount lustre-MDT0001 complete [ 7254.576767] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 7255.145785] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 18:20:09 (1786486809) [ 7257.502304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7257.688536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7258.989876] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7261.791196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7261.909944] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7263.248109] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7265.681726] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7267.513424] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7269.817074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7269.930196] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7271.846311] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7274.216268] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7274.878882] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 7285.602923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 7287.264753] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7287.267009] LustreError: Skipped 1 previous similar message [ 7287.268776] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7287.273933] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7288.290205] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7293.068355] Lustre: server umount lustre-OST0000 complete [ 7298.016793] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7298.019489] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7298.024798] Lustre: Skipped 1 previous similar message [ 7298.027297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7298.029030] Lustre: Skipped 2 previous similar messages [ 7300.375303] Lustre: server umount lustre-MDT0000 complete [ 7301.641506] LustreError: 407022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786486856 with bad export cookie 2209327336057505982 [ 7301.644233] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7301.646603] LustreError: 407022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7301.765601] Lustre: server umount lustre-MDT0001 complete [ 7305.303775] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 7306.405441] Key type lgssc unregistered [ 7306.537372] LNet: 410786:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7306.541363] LNetError: 410786:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7306.550284] LNet: Removed LNI 192.168.203.122@tcp [ 7306.861875] Key type .llcrypt unregistered [ 7306.863412] Key type ._llcrypt unregistered [ 7314.444327] Key type ._llcrypt registered [ 7314.445351] Key type .llcrypt registered [ 7314.479758] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 7314.590393] Key type .llcrypt unregistered [ 7314.591612] Key type ._llcrypt unregistered [ 7321.731343] Key type ._llcrypt registered [ 7321.733205] Key type .llcrypt registered [ 7321.771568] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 7322.112941] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7322.172607] alg: No test for adler32 (adler32-zlib) [ 7323.019615] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 7323.111378] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 7324.688201] Key type lgssc registered [ 7325.049636] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7327.725981] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7327.733228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7328.877758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7330.217283] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7333.251044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7333.378688] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7335.393692] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7337.864436] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7338.464874] LustreError: 412976:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7338.470000] LustreError: 412976:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 7339.492155] LustreError: 412975:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7339.496217] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7339.498498] LustreError: 412975:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7340.112145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7340.234798] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7341.541146] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7349.728740] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7349.731022] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7349.734906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7350.758747] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7350.758884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7350.762075] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7354.705833] Lustre: server umount lustre-OST0000 complete [ 7355.874496] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7355.874792] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7355.875120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7355.875125] Lustre: Skipped 1 previous similar message [ 7355.879130] Lustre: Skipped 1 previous similar message [ 7360.993231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7360.995303] Lustre: Skipped 2 previous similar messages [ 7362.016066] Lustre: server umount lustre-MDT0000 complete [ 7363.262027] LustreError: 412956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786486917 with bad export cookie 10317564415444111080 [ 7363.264553] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7363.265698] LustreError: 412956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7363.393899] Lustre: server umount lustre-MDT0001 complete [ 7366.828126] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 7367.894501] Key type lgssc unregistered [ 7368.017356] LNet: 416124:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7368.019849] LNetError: 416124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7368.027321] LNet: Removed LNI 192.168.203.122@tcp [ 7368.342524] Key type .llcrypt unregistered [ 7368.343763] Key type ._llcrypt unregistered [ 7377.075169] Key type ._llcrypt registered [ 7377.076087] Key type .llcrypt registered [ 7377.109503] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 7377.418879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7377.449556] alg: No test for adler32 (adler32-zlib) [ 7378.291164] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 7378.370947] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 7379.952155] Key type lgssc registered [ 7380.290474] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7382.720087] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7382.726150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7398.176319] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7403.296257] LustreError: 417490:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.122@tcp: failed processing log, type 4: rc = -110 [ 7429.920189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7431.858365] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7434.258830] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7436.523861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7436.700883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7436.727076] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 7438.044898] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7440.966437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7441.099655] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7442.500167] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7445.027566] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7446.965024] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7450.595771] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7450.596898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7450.602438] Lustre: Skipped 1 previous similar message [ 7455.447856] Lustre: server umount lustre-OST0000 complete [ 7459.810296] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 7459.812376] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7459.816071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7459.818257] Lustre: Skipped 1 previous similar message [ 7460.833208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7460.833458] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7462.878279] Lustre: server umount lustre-MDT0000 complete [ 7464.127437] LustreError: 417497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786487018 with bad export cookie 15231399078149050403 [ 7464.130825] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7464.131435] LustreError: 417497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7464.249196] Lustre: server umount lustre-MDT0001 complete [ 7467.842134] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 7468.762988] Key type lgssc unregistered [ 7468.881313] LNet: 421061:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7468.883636] LNetError: 421061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7468.892312] LNet: Removed LNI 192.168.203.122@tcp [ 7469.175325] Key type .llcrypt unregistered [ 7469.176875] Key type ._llcrypt unregistered [ 7477.582811] Key type ._llcrypt registered [ 7477.584402] Key type .llcrypt registered [ 7477.616529] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 7478.028337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7478.039376] alg: No test for adler32 (adler32-zlib) [ 7478.897070] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 7478.993262] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 7480.576144] Key type lgssc registered [ 7480.957532] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7483.627121] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7483.635045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7484.802564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7486.258692] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7489.231086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7489.353918] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7490.767242] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7493.199866] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7495.088379] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7497.389128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7497.508269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7499.495260] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7501.925570] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7510.828784] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 7515.105252] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7515.107396] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7515.110931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7516.129229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7516.131326] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7517.462943] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.22@tcp (stopping) [ 7517.465996] Lustre: Skipped 2 previous similar messages [ 7519.449588] Lustre: server umount lustre-MDT0000 complete [ 7520.242802] LustreError: 422435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.22@tcp arrived at 1786487074 with bad export cookie 8470182320681757909 [ 7520.248794] LustreError: 422435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7520.904569] LustreError: 423189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786487075 with bad export cookie 8470182320681757286 [ 7520.908432] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7521.248909] LustreError: 422472:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7521.249595] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7521.254389] LustreError: 422472:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7521.259591] Lustre: Skipped 1 previous similar message [ 7521.260292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7525.280899] LustreError: 422454:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7525.281280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7525.286575] LustreError: 422454:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7526.995558] Lustre: server umount lustre-OST0000 complete [ 7529.388670] Lustre: server umount lustre-MDT0001 complete [ 7533.023518] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 7534.132251] Key type lgssc unregistered [ 7534.264414] LNet: 426158:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7534.266836] LNetError: 426158:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7534.278275] LNet: Removed LNI 192.168.203.122@tcp [ 7534.611569] Key type .llcrypt unregistered [ 7534.614270] Key type ._llcrypt unregistered [ 7543.696268] Key type ._llcrypt registered [ 7543.697856] Key type .llcrypt registered [ 7543.736144] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 7544.189233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7544.253181] alg: No test for adler32 (adler32-zlib) [ 7545.117895] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 7545.216089] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 7546.800112] Key type lgssc registered [ 7547.163935] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7549.716356] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7549.724411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7550.876106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7552.336868] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7555.034343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7555.152900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7556.466714] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7558.774399] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7560.502089] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7562.706988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7562.825326] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7564.669897] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7567.040890] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7576.166316] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 7580.640735] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 7580.643200] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7580.647253] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7581.666391] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7581.668797] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7584.338664] Lustre: server umount lustre-MDT0001 complete [ 7586.786679] LustreError: 429837:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7586.786890] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7586.787342] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7586.787912] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7586.787916] Lustre: Skipped 2 previous similar messages [ 7586.792123] LustreError: 429837:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7591.905041] LustreError: 427552:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7591.905702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7592.080698] Lustre: server umount lustre-OST0000 complete [ 7593.398611] Lustre: server umount lustre-MDT0000 complete [ 7597.832166] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 7598.839976] Key type lgssc unregistered [ 7598.973426] LNet: 431258:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7598.975909] LNetError: 431258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7598.989287] LNet: Removed LNI 192.168.203.122@tcp [ 7599.259351] Key type .llcrypt unregistered [ 7599.260380] Key type ._llcrypt unregistered [ 7607.079352] Key type ._llcrypt registered [ 7607.080325] Key type .llcrypt registered [ 7607.108471] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 7612.939869] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 7613.355675] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7613.365693] alg: No test for adler32 (adler32-zlib) [ 7614.217598] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 7614.300601] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 7615.880132] Key type lgssc registered [ 7616.217749] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7628.242079] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 7633.668385] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7633.676154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7634.755659] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7634.765810] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7634.804208] Lustre: lustre-MDT0000: new disk, initializing [ 7634.827561] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7634.833607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7636.197837] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7641.939690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7642.037338] Lustre: lustre-OST0000: new disk, initializing [ 7642.040114] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7642.042346] Lustre: Skipped 1 previous similar message [ 7642.044829] Lustre: 436260:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7642.072189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7644.038047] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7647.778575] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7649.281101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 7649.285671] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 7649.293556] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 7658.485853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7658.529561] Lustre: 435244:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7658.558211] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7658.593824] Lustre: lustre-MDT0001: new disk, initializing [ 7658.635700] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7658.649950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 7658.655636] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 7660.010161] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7663.661454] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7663.735767] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7665.090242] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7665.169422] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7669.017136] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.22@tcp (stopping) [ 7669.730260] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 7669.731209] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7669.733665] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7669.736979] Lustre: Skipped 1 previous similar message [ 7672.921840] Lustre: server umount lustre-MDT0001 complete [ 7679.971721] LustreError: 435250:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7679.979086] LustreError: 435250:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7685.093448] LustreError: 436268:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7685.098164] LustreError: 436268:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7690.209954] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7690.210642] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7690.210823] LustreError: 435255:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7690.226899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7690.228995] Lustre: Skipped 1 previous similar message [ 7691.481498] Lustre: server umount lustre-OST0000 complete [ 7699.125740] Lustre: server umount lustre-MDT0000 complete [ 7704.074654] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 7705.165349] Key type lgssc unregistered [ 7705.307333] LNet: 440011:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7705.310871] LNetError: 440011:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7705.318358] LNet: Removed LNI 192.168.203.122@tcp [ 7705.652567] Key type .llcrypt unregistered [ 7705.653875] Key type ._llcrypt unregistered [ 7717.880980] Key type ._llcrypt registered [ 7717.882749] Key type .llcrypt registered [ 7717.920034] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 7718.023204] Key type .llcrypt unregistered [ 7718.024078] Key type ._llcrypt unregistered [ 7724.191989] Key type ._llcrypt registered [ 7724.192980] Key type .llcrypt registered [ 7724.227980] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 7724.658482] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7724.667499] alg: No test for adler32 (adler32-zlib) [ 7725.497882] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 7725.572704] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 7727.152141] Key type lgssc registered [ 7727.489020] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7735.851589] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7735.858902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7736.930086] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7736.940844] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7736.974552] Lustre: lustre-MDT0000: new disk, initializing [ 7736.992183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7736.997794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7738.267657] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7742.230660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7742.268200] Lustre: 443362:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7742.289228] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7742.290880] Lustre: Skipped 1 previous similar message [ 7742.322569] Lustre: lustre-MDT0001: new disk, initializing [ 7742.352422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7742.365209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7742.368528] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7743.658395] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7747.240883] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7749.004697] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7751.309804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7751.412617] Lustre: lustre-OST0000: new disk, initializing [ 7751.414310] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7751.416835] Lustre: 445330:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7751.448038] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7751.725207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7751.728418] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7751.760376] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7753.384903] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7757.112350] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7761.890745] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7761.892081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7761.897112] Lustre: Skipped 2 previous similar messages [ 7761.898967] Lustre: Skipped 1 previous similar message [ 7765.662144] Lustre: server umount lustre-MDT0000 complete [ 7766.907134] LustreError: 443355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786487321 with bad export cookie 17350730915624692736 [ 7766.910835] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7766.911503] LustreError: 443355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7767.030577] Lustre: server umount lustre-MDT0001 complete [ 7778.519274] Lustre: server umount lustre-OST0000 complete [ 7782.032881] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 18:28:56 (1786487336) [ 7784.668191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7784.851579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7786.280511] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7789.204066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7789.320900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7790.726159] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7793.229057] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7795.161455] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7810.594960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7810.665394] Lustre: Found index 0 for lustre-OST0000, updating log [ 7810.730667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7811.751364] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 7812.764060] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7815.252426] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7817.184909] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7817.186915] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7817.190343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7817.192222] Lustre: Skipped 1 previous similar message [ 7819.235031] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7819.238155] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7819.243769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7822.421235] Lustre: server umount lustre-OST0000 complete [ 7834.997735] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7834.998472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7835.001930] Lustre: Skipped 1 previous similar message [ 7835.004952] Lustre: Skipped 2 previous similar messages [ 7840.225064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7840.227037] Lustre: Skipped 2 previous similar messages [ 7841.175432] Lustre: server umount lustre-MDT0000 complete [ 7842.376290] LustreError: 447338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786487396 with bad export cookie 17350730915624694115 [ 7842.379652] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7842.381399] LustreError: 447338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7842.506767] Lustre: server umount lustre-MDT0001 complete [ 7845.812336] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 18:29:59 (1786487399) [ 7874.080282] LNet: 1 local NIs in recovery (showing 1): 192.168.203.122@tcp [ 8425.823020] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8425.972751] LustreError: 453759:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8425.981168] LustreError: 453759:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8425.997608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8427.383093] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8430.199419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8431.634477] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8434.060351] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8435.941610] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8451.235451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8451.346831] Lustre: Found index 0 for lustre-OST0000, updating log [ 8453.374091] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8454.503898] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [ 8455.689512] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8457.378044] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8457.382612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8462.484648] Lustre: server umount lustre-OST0000 complete [ 8464.866888] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8464.870025] Lustre: Skipped 1 previous similar message [ 8474.081334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8474.083631] Lustre: Skipped 21 previous similar messages [ 8478.176211] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8478.245407] Lustre: server umount lustre-MDT0000 complete [ 8479.201159] LustreError: 454488:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8479.206259] LustreError: 454488:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8479.596729] LustreError: 455662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786488034 with bad export cookie 17350730915624695186 [ 8479.602559] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8479.603273] LustreError: 455662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8479.727752] Lustre: server umount lustre-MDT0001 complete [ 8488.480241] LNet: 1 local NIs in recovery (showing 1): 192.168.203.122@tcp [ 8497.231670] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 8500.585036] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 8512.988410] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 18:41:07 (1786488067) [ 8518.822631] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 8519.856080] Key type lgssc unregistered [ 8519.980408] LNet: 460417:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8519.984163] LNetError: 460417:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8582.437558] LNet: Removed LNI 192.168.203.122@tcp [ 8582.832655] Key type .llcrypt unregistered [ 8582.834701] Key type ._llcrypt unregistered [ 8589.955008] Key type ._llcrypt registered [ 8589.956775] Key type .llcrypt registered [ 8590.005667] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 8590.422396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8590.482295] alg: No test for adler32 (adler32-zlib) [ 8591.373211] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 8591.484149] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 8593.072163] Key type lgssc registered [ 8593.518125] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8594.978416] LNet: Added LNI 192.168.203.122@tcp100 [8/256/0/180] [ 8595.290719] LNet: Added LNI 192.168.203.122@tcp101 [8/256/0/180] [ 8595.626378] LNet: Added LNI 192.168.203.122@tcp102 [8/256/0/180] [ 8595.959781] LNet: Added LNI 192.168.203.122@tcp103 [8/256/0/180] [ 8596.265825] LNet: Added LNI 192.168.203.122@tcp104 [8/256/0/180] [ 8596.568541] LNet: Added LNI 192.168.203.122@tcp105 [8/256/0/180] [ 8596.869302] LNet: Added LNI 192.168.203.122@tcp106 [8/256/0/180] [ 8597.168176] LNet: Added LNI 192.168.203.122@tcp107 [8/256/0/180] [ 8597.461620] LNet: Added LNI 192.168.203.122@tcp108 [8/256/0/180] [ 8597.750274] LNet: Added LNI 192.168.203.122@tcp109 [8/256/0/180] [ 8598.034923] LNet: Added LNI 192.168.203.122@tcp110 [8/256/0/180] [ 8598.325358] LNet: Added LNI 192.168.203.122@tcp111 [8/256/0/180] [ 8598.610868] LNet: Added LNI 192.168.203.122@tcp112 [8/256/0/180] [ 8598.901780] LNet: Added LNI 192.168.203.122@tcp113 [8/256/0/180] [ 8599.191275] LNet: Added LNI 192.168.203.122@tcp114 [8/256/0/180] [ 8599.474246] LNet: Added LNI 192.168.203.122@tcp115 [8/256/0/180] [ 8599.755784] LNet: Added LNI 192.168.203.122@tcp116 [8/256/0/180] [ 8600.044906] LNet: Added LNI 192.168.203.122@tcp117 [8/256/0/180] [ 8600.327645] LNet: Added LNI 192.168.203.122@tcp118 [8/256/0/180] [ 8600.617568] LNet: Added LNI 192.168.203.122@tcp119 [8/256/0/180] [ 8600.906294] LNet: Added LNI 192.168.203.122@tcp120 [8/256/0/180] [ 8601.202536] LNet: Added LNI 192.168.203.122@tcp121 [8/256/0/180] [ 8601.496051] LNet: Added LNI 192.168.203.122@tcp122 [8/256/0/180] [ 8601.775184] LNet: Added LNI 192.168.203.122@tcp123 [8/256/0/180] [ 8602.083504] LNet: Added LNI 192.168.203.122@tcp124 [8/256/0/180] [ 8602.408565] LNet: Added LNI 192.168.203.122@tcp125 [8/256/0/180] [ 8602.719933] LNet: Added LNI 192.168.203.122@tcp126 [8/256/0/180] [ 8603.031752] LNet: Added LNI 192.168.203.122@tcp127 [8/256/0/180] [ 8603.375048] LNet: Added LNI 192.168.203.122@tcp128 [8/256/0/180] [ 8603.684638] LNet: Added LNI 192.168.203.122@tcp129 [8/256/0/180] [ 8603.969313] LNet: Added LNI 192.168.203.122@tcp130 [8/256/0/180] [ 8604.266932] LNet: Added LNI 192.168.203.122@tcp131 [8/256/0/180] [ 8604.550435] LNet: Added LNI 192.168.203.122@tcp132 [8/256/0/180] [ 8604.838818] LNet: Added LNI 192.168.203.122@tcp133 [8/256/0/180] [ 8605.135029] LNet: Added LNI 192.168.203.122@tcp134 [8/256/0/180] [ 8605.419859] LNet: Added LNI 192.168.203.122@tcp135 [8/256/0/180] [ 8607.569043] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8607.573636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8608.610925] Lustre: 463415:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.122@tcp: can't add NID 192.168.203.122@tcp130: rc = -75 [ 8608.637378] LNetError: 460877:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.122 [ 8608.642151] LNetError: Protocol error connecting to 192.168.203.122@tcp119 at host 192.168.203.122:988: is it running a compatible version of Lustre? [ 8608.645343] LNetError: Skipped 3 previous similar messages [ 8608.698640] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8608.711413] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8608.748569] Lustre: lustre-MDT0000: new disk, initializing [ 8608.770757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8608.776920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8608.928129] LNet: 11 local NIs in recovery (showing 11): 192.168.203.122@tcp124, 192.168.203.122@tcp125, 192.168.203.122@tcp119, 192.168.203.122@tcp127, 192.168.203.122@tcp126, 192.168.203.122@tcp128, 192.168.203.122@tcp129, 192.168.203.122@tcp130, 192.168.203.122@tcp131, 192.168.203.122@tcp132, 192.168.203.122@tcp133 [ 8609.698651] LNetError: 463444:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.122 [ 8609.702711] LNetError: 463444:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 15 previous similar messages [ 8609.705555] LNetError: Protocol error connecting to 192.168.203.122@tcp120 at host 192.168.203.122:988: is it running a compatible version of Lustre? [ 8609.709341] LNetError: Skipped 11 previous similar messages [ 8610.061452] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8612.002866] LNetError: 463443:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.122 [ 8612.008559] LNetError: 463443:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 10 previous similar messages [ 8612.018439] LNetError: Protocol error connecting to 192.168.203.122@tcp125 at host 192.168.203.122:988: is it running a compatible version of Lustre? [ 8612.024286] LNetError: Skipped 13 previous similar messages [ 8614.114107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8614.166650] Lustre: 463432:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8614.191815] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8614.193910] Lustre: Skipped 1 previous similar message [ 8614.231646] Lustre: lustre-MDT0001: new disk, initializing [ 8614.240239] LNetError: 460876:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.122 [ 8614.243740] LNetError: 460876:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 3 previous similar messages [ 8614.243872] Lustre: 464235:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.122@tcp: can't add NID 192.168.203.122@tcp130: rc = -75 [ 8614.248020] LNetError: Protocol error connecting to 192.168.203.122@tcp134 at host 192.168.203.122:988: is it running a compatible version of Lustre? [ 8614.250545] Lustre: 464235:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 8614.298454] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8614.310134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8614.313199] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8615.619675] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8615.673143] Lustre: 464504:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.122@tcp: can't add NID 192.168.203.122@tcp130: rc = -75 [ 8615.676016] Lustre: 464504:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 8619.291486] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8621.219330] LNetError: 463439:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.122 [ 8621.223518] LNetError: 463439:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 5 previous similar messages [ 8621.226382] LNetError: Protocol error connecting to 192.168.203.122@tcp135 at host 192.168.203.122:988: is it running a compatible version of Lustre? [ 8621.228222] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8621.230181] LNetError: Skipped 5 previous similar messages [ 8623.941289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8624.096772] Lustre: lustre-OST0000: new disk, initializing [ 8624.099538] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8624.104505] Lustre: 465407:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8624.164411] Lustre: 465412:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.122@tcp: can't add NID 192.168.203.122@tcp130: rc = -75 [ 8624.169143] Lustre: 465412:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [ 8624.192751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8626.440292] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8629.409186] LNetError: 463443:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.122 [ 8629.412250] LNetError: 463443:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 1 previous similar message [ 8629.414490] LNetError: Protocol error connecting to 192.168.203.122@tcp135 at host 192.168.203.122:988: is it running a compatible version of Lustre? [ 8629.417519] LNetError: Skipped 1 previous similar message [ 8629.745902] Lustre: 466069:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.122@tcp: can't add NID 192.168.203.122@tcp130: rc = -75 [ 8629.749167] Lustre: 466069:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 8629.764888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8629.768721] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8629.822467] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8630.327051] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8634.854429] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8634.855483] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8634.858511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8634.872426] Lustre: Skipped 1 previous similar message [ 8638.864934] Lustre: server umount lustre-OST0000 complete [ 8645.090917] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8645.091582] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8645.091879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8645.091884] Lustre: Skipped 1 previous similar message [ 8645.094979] Lustre: Skipped 1 previous similar message [ 8645.098254] LustreError: Skipped 1 previous similar message [ 8645.793198] LNetError: 460879:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.122 [ 8645.796205] LNetError: 460879:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 5 previous similar messages [ 8645.798749] LNetError: Protocol error connecting to 192.168.203.122@tcp135 at host 192.168.203.122:988: is it running a compatible version of Lustre? [ 8645.801753] LNetError: Skipped 5 previous similar messages [ 8646.234757] Lustre: server umount lustre-MDT0000 complete [ 8647.437183] LustreError: 463423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786488201 with bad export cookie 10442441190924659737 [ 8647.440030] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8647.441115] LustreError: 463423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8647.560051] Lustre: server umount lustre-MDT0001 complete [ 8651.183668] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 8652.187050] Key type lgssc unregistered [ 8652.344277] LNet: 467324:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8652.346925] LNetError: 467324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8652.348892] LNet: Removed LNI 192.168.203.122@tcp [ 8653.408165] LNet: Removed LNI 192.168.203.122@tcp100 [ 8653.409649] LNetError: 467324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8653.412450] LNetError: 467324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 8653.414640] LNet: Removed LNI 192.168.203.122@tcp101 [ 8654.432118] LNet: Removed LNI 192.168.203.122@tcp102 [ 8654.433507] LNetError: 467324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8654.435643] LNetError: 467324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 8654.438437] LNet: Removed LNI 192.168.203.122@tcp103 [ 8655.456153] LNet: Removed LNI 192.168.203.122@tcp104 [ 8656.480144] LNet: Removed LNI 192.168.203.122@tcp105 [ 8656.481530] LNetError: 467324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8656.483706] LNetError: 467324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 8656.486034] LNet: Removed LNI 192.168.203.122@tcp106 [ 8657.504171] LNet: Removed LNI 192.168.203.122@tcp107 [ 8658.528119] LNet: Removed LNI 192.168.203.122@tcp108 [ 8659.552202] LNet: Removed LNI 192.168.203.122@tcp109 [ 8660.576108] LNet: Removed LNI 192.168.203.122@tcp110 [ 8660.577595] LNetError: 467324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8660.579992] LNetError: 467324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 8660.582571] LNet: Removed LNI 192.168.203.122@tcp111 [ 8661.600159] LNet: Removed LNI 192.168.203.122@tcp112 [ 8662.624114] LNet: Removed LNI 192.168.203.122@tcp113 [ 8663.648138] LNet: Removed LNI 192.168.203.122@tcp114 [ 8664.672115] LNet: Removed LNI 192.168.203.122@tcp115 [ 8665.696122] LNet: Removed LNI 192.168.203.122@tcp116 [ 8666.720162] LNet: Removed LNI 192.168.203.122@tcp117 [ 8667.744193] LNet: Removed LNI 192.168.203.122@tcp118 [ 8668.768268] LNet: Removed LNI 192.168.203.122@tcp119 [ 8668.769841] LNetError: 467324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8668.771786] LNetError: 467324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 8668.774918] LNet: Removed LNI 192.168.203.122@tcp120 [ 8669.792240] LNet: Removed LNI 192.168.203.122@tcp121 [ 8670.816181] LNet: Removed LNI 192.168.203.122@tcp122 [ 8671.840129] LNet: Removed LNI 192.168.203.122@tcp123 [ 8672.864148] LNet: Removed LNI 192.168.203.122@tcp124 [ 8672.865613] LNet: Removed LNI 192.168.203.122@tcp125 [ 8673.888138] LNet: Removed LNI 192.168.203.122@tcp126 [ 8674.912163] LNet: Removed LNI 192.168.203.122@tcp127 [ 8675.936132] LNet: Removed LNI 192.168.203.122@tcp128 [ 8676.960165] LNet: Removed LNI 192.168.203.122@tcp129 [ 8677.984172] LNet: Removed LNI 192.168.203.122@tcp130 [ 8679.008166] LNet: Removed LNI 192.168.203.122@tcp131 [ 8680.032147] LNet: Removed LNI 192.168.203.122@tcp132 [ 8681.056177] LNet: Removed LNI 192.168.203.122@tcp133 [ 8682.080097] LNet: Removed LNI 192.168.203.122@tcp134 [ 8682.087530] LNet: Removed LNI 192.168.203.122@tcp135 [ 8682.388098] Key type .llcrypt unregistered [ 8682.389174] Key type ._llcrypt unregistered [ 8689.924665] Key type ._llcrypt registered [ 8689.926122] Key type .llcrypt registered [ 8689.962752] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 8696.053353] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 8696.446783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8696.460665] alg: No test for adler32 (adler32-zlib) [ 8697.314941] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 8697.415799] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 8699.008203] Key type lgssc registered [ 8699.374422] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8710.426165] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 18:44:24 (1786488264) [ 8716.176958] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 8717.216737] Key type lgssc unregistered [ 8717.337497] LNet: 471386:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8717.340723] LNetError: 471386:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8717.353493] LNet: Removed LNI 192.168.203.122@tcp [ 8717.650323] Key type .llcrypt unregistered [ 8717.651326] Key type ._llcrypt unregistered [ 8724.112183] Key type ._llcrypt registered [ 8724.113262] Key type .llcrypt registered [ 8724.153501] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 8724.503798] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8724.557053] alg: No test for adler32 (adler32-zlib) [ 8725.425835] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 8725.521924] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 8727.104101] Key type lgssc registered [ 8727.419807] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8730.410966] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8730.415865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8731.500882] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8731.510597] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8731.550545] Lustre: lustre-MDT0000: new disk, initializing [ 8731.570426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8731.575973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8732.884329] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8737.045586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8737.100402] Lustre: 472574:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8737.122622] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8737.124614] Lustre: Skipped 1 previous similar message [ 8737.163472] Lustre: lustre-MDT0001: new disk, initializing [ 8737.194907] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8737.207407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8737.210799] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8738.597785] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8742.226753] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8744.047202] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8746.367328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8746.494234] Lustre: lustre-OST0000: new disk, initializing [ 8746.496538] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8746.499441] Lustre: 474539:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8746.533262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8747.261466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8747.264743] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8747.304321] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8748.589520] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8752.297730] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8753.449528] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing lnet_if_list [ 8755.066485] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 8755.081990] LNet: Added LNI 192.168.203.122@tcp10 [8/256/0/180] [ 8756.768818] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 8756.778994] LNet: Added LNI 192.168.203.122@tcp11 [8/256/0/180] [ 8758.068093] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 8758.079030] LNet: Added LNI 192.168.203.122@tcp12 [8/256/0/180] [ 8759.486801] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 8759.502310] LNet: Added LNI 192.168.203.122@tcp13 [8/256/0/180] [ 8760.980022] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 8760.991875] LNet: Added LNI 192.168.203.122@tcp14 [8/256/0/180] [ 8762.355323] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 8762.365700] LNet: Added LNI 192.168.203.122@tcp15 [8/256/0/180] [ 8763.799614] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 8763.813880] LNet: Added LNI 192.168.203.122@tcp16 [8/256/0/180] [ 8765.238505] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 8765.252143] LNet: Added LNI 192.168.203.122@tcp17 [8/256/0/180] [ 8766.755565] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 8766.766602] LNet: Added LNI 192.168.203.122@tcp18 [8/256/0/180] [ 8768.238822] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 8768.249076] LNet: Added LNI 192.168.203.122@tcp19 [8/256/0/180] [ 8769.724401] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 8769.735960] LNet: Added LNI 192.168.203.122@tcp20 [8/256/0/180] [ 8771.275881] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 8771.285549] LNet: Added LNI 192.168.203.122@tcp21 [8/256/0/180] [ 8772.936348] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 8772.952460] LNet: Added LNI 192.168.203.122@tcp22 [8/256/0/180] [ 8774.601837] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 8774.612286] LNet: Added LNI 192.168.203.122@tcp23 [8/256/0/180] [ 8776.502259] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 8776.515137] LNet: Added LNI 192.168.203.122@tcp24 [8/256/0/180] [ 8778.341599] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 8778.354872] LNet: Added LNI 192.168.203.122@tcp25 [8/256/0/180] [ 8780.098571] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 8780.109881] LNet: Added LNI 192.168.203.122@tcp26 [8/256/0/180] [ 8781.897862] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 8781.907173] LNet: Added LNI 192.168.203.122@tcp27 [8/256/0/180] [ 8783.777050] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 8783.790415] LNet: Added LNI 192.168.203.122@tcp28 [8/256/0/180] [ 8785.608249] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 8785.619130] LNet: Added LNI 192.168.203.122@tcp29 [8/256/0/180] [ 8787.455165] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 8787.467945] LNet: Added LNI 192.168.203.122@tcp30 [8/256/0/180] [ 8789.345015] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 8789.354258] LNet: Added LNI 192.168.203.122@tcp31 [8/256/0/180] [ 8791.276910] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 8791.287806] LNet: Added LNI 192.168.203.122@tcp32 [8/256/0/180] [ 8793.205192] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 8793.216344] LNet: Added LNI 192.168.203.122@tcp33 [8/256/0/180] [ 8795.353415] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 8795.362798] LNet: Added LNI 192.168.203.122@tcp34 [8/256/0/180] [ 8797.325410] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 8797.337767] LNet: Added LNI 192.168.203.122@tcp35 [8/256/0/180] [ 8799.325480] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 8799.338663] LNet: Added LNI 192.168.203.122@tcp36 [8/256/0/180] [ 8801.390630] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 8801.400377] LNet: Added LNI 192.168.203.122@tcp37 [8/256/0/180] [ 8803.539562] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 8803.550611] LNet: Added LNI 192.168.203.122@tcp38 [8/256/0/180] [ 8805.778423] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 8805.788889] LNet: Added LNI 192.168.203.122@tcp39 [8/256/0/180] [ 8807.952801] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 8807.962757] LNet: Added LNI 192.168.203.122@tcp40 [8/256/0/180] [ 8810.266696] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 8810.278319] LNet: Added LNI 192.168.203.122@tcp41 [8/256/0/180] [ 8812.583593] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 8812.595295] LNet: Added LNI 192.168.203.122@tcp42 [8/256/0/180] [ 8814.919652] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 8814.932333] LNet: Added LNI 192.168.203.122@tcp43 [8/256/0/180] [ 8817.355437] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 8817.367498] LNet: Added LNI 192.168.203.122@tcp44 [8/256/0/180] [ 8819.744163] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 8819.754113] LNet: Added LNI 192.168.203.122@tcp45 [8/256/0/180] [ 8822.144785] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 8822.156418] LNet: Added LNI 192.168.203.122@tcp46 [8/256/0/180] [ 8824.557537] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 8824.571316] LNet: Added LNI 192.168.203.122@tcp47 [8/256/0/180] [ 8827.083064] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 8827.093225] LNet: Added LNI 192.168.203.122@tcp48 [8/256/0/180] [ 8829.583094] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 8829.593544] LNet: Added LNI 192.168.203.122@tcp49 [8/256/0/180] [ 8832.234358] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 8832.246226] LNet: Added LNI 192.168.203.122@tcp50 [8/256/0/180] [ 8844.771397] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8844.772086] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8844.773103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8844.779668] Lustre: Skipped 1 previous similar message [ 8845.923650] Lustre: server umount lustre-MDT0000 complete [ 8847.093163] LustreError: 474538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786488401 with bad export cookie 11873303901922294418 [ 8847.096686] LustreError: 474538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8847.096928] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8847.205586] Lustre: server umount lustre-MDT0001 complete [ 8854.258462] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 8857.031905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8866.208117] Lustre: 471850:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786488404/real 1786488404] req@00000000f19b2dd5 x1873268733721856/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786488420 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8866.216853] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8866.220578] Lustre: Skipped 1 previous similar message [ 8869.344152] Lustre: 471848:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786488407/real 1786488407] req@000000007e812ce5 x1873268733721984/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786488423 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8871.392152] Lustre: 471848:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786488409/real 1786488409] req@00000000eca991ea x1873268733722112/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786488425 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8872.352428] LustreError: 471847:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000e7119aa7 x1873268733723008/t0(0) o250->MGC192.168.203.122@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8872.463666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8873.504141] Lustre: 471850:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786488411/real 1786488411] req@000000006fa223a7 x1873268733722368/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786488427 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8873.867867] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8876.656904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8876.785096] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8877.752072] Lustre: 471851:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786488416/real 1786488416] req@000000000834772d x1873268733722496/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786488432 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8878.133038] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8880.607767] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8882.149715] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8882.412026] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8885.221673] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8890.338909] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8890.340195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8890.342750] Lustre: Skipped 1 previous similar message [ 8890.344538] Lustre: Skipped 2 previous similar messages [ 8895.459475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8895.461509] Lustre: Skipped 2 previous similar messages [ 8895.956573] Lustre: server umount lustre-OST0000 complete [ 8897.504876] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8897.507330] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8897.511253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8903.257100] Lustre: server umount lustre-MDT0000 complete [ 8904.482049] LustreError: 473368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786488458 with bad export cookie 11873303901922296455 [ 8904.485600] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8904.485965] LustreError: 473368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8904.596833] Lustre: server umount lustre-MDT0001 complete [ 8907.973324] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 8908.989702] Key type lgssc unregistered [ 8909.137034] LNet: 501504:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8909.140570] LNetError: 501504:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8909.143994] LNet: Removed LNI 192.168.203.122@tcp [ 8909.145314] LNet: Removed LNI 192.168.203.122@tcp10 [ 8909.146623] LNet: Removed LNI 192.168.203.122@tcp11 [ 8909.147835] LNet: Removed LNI 192.168.203.122@tcp12 [ 8909.149181] LNet: Removed LNI 192.168.203.122@tcp13 [ 8909.150470] LNet: Removed LNI 192.168.203.122@tcp14 [ 8909.151647] LNet: Removed LNI 192.168.203.122@tcp15 [ 8909.152812] LNet: Removed LNI 192.168.203.122@tcp16 [ 8909.153987] LNet: Removed LNI 192.168.203.122@tcp17 [ 8909.155186] LNet: Removed LNI 192.168.203.122@tcp18 [ 8909.156493] LNet: Removed LNI 192.168.203.122@tcp19 [ 8909.157752] LNet: Removed LNI 192.168.203.122@tcp20 [ 8909.159065] LNet: Removed LNI 192.168.203.122@tcp21 [ 8909.160407] LNet: Removed LNI 192.168.203.122@tcp22 [ 8909.161677] LNet: Removed LNI 192.168.203.122@tcp23 [ 8909.162884] LNet: Removed LNI 192.168.203.122@tcp24 [ 8909.164035] LNet: Removed LNI 192.168.203.122@tcp25 [ 8909.165270] LNet: Removed LNI 192.168.203.122@tcp26 [ 8909.166489] LNet: Removed LNI 192.168.203.122@tcp27 [ 8909.167750] LNet: Removed LNI 192.168.203.122@tcp28 [ 8909.168927] LNet: Removed LNI 192.168.203.122@tcp29 [ 8909.170114] LNet: Removed LNI 192.168.203.122@tcp30 [ 8909.171366] LNet: Removed LNI 192.168.203.122@tcp31 [ 8909.172531] LNet: Removed LNI 192.168.203.122@tcp32 [ 8909.173664] LNet: Removed LNI 192.168.203.122@tcp33 [ 8909.174835] LNet: Removed LNI 192.168.203.122@tcp34 [ 8909.175960] LNet: Removed LNI 192.168.203.122@tcp35 [ 8909.177216] LNet: Removed LNI 192.168.203.122@tcp36 [ 8909.178425] LNet: Removed LNI 192.168.203.122@tcp37 [ 8909.179648] LNet: Removed LNI 192.168.203.122@tcp38 [ 8909.180792] LNet: Removed LNI 192.168.203.122@tcp39 [ 8909.181925] LNet: Removed LNI 192.168.203.122@tcp40 [ 8909.183025] LNet: Removed LNI 192.168.203.122@tcp41 [ 8909.184148] LNet: Removed LNI 192.168.203.122@tcp42 [ 8909.185373] LNet: Removed LNI 192.168.203.122@tcp43 [ 8909.186539] LNet: Removed LNI 192.168.203.122@tcp44 [ 8909.187684] LNet: Removed LNI 192.168.203.122@tcp45 [ 8909.188918] LNet: Removed LNI 192.168.203.122@tcp46 [ 8909.190156] LNet: Removed LNI 192.168.203.122@tcp47 [ 8909.191359] LNet: Removed LNI 192.168.203.122@tcp48 [ 8909.192470] LNet: Removed LNI 192.168.203.122@tcp49 [ 8909.203391] LNet: Removed LNI 192.168.203.122@tcp50 [ 8909.458384] Key type .llcrypt unregistered [ 8909.459599] Key type ._llcrypt unregistered [ 8915.884180] Key type ._llcrypt registered [ 8915.885289] Key type .llcrypt registered [ 8915.927638] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 8922.090357] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 8922.450712] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8922.498261] alg: No test for adler32 (adler32-zlib) [ 8923.350226] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 8923.436392] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 8925.016130] Key type lgssc registered [ 8925.370241] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8936.702522] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8936.708090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8937.788942] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8937.798893] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8937.840582] Lustre: lustre-MDT0000: new disk, initializing [ 8937.859805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8937.865308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8939.129161] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8943.118513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8943.164170] Lustre: 504672:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8943.191867] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8943.194521] Lustre: Skipped 1 previous similar message [ 8943.232356] Lustre: lustre-MDT0001: new disk, initializing [ 8943.267510] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8943.278758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8943.282231] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8944.714735] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8948.313245] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8950.122650] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8952.530932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8952.604913] Lustre: *** cfs_fail_loc=90f, val=0*** [ 8952.632335] Lustre: lustre-OST0000: new disk, initializing [ 8952.634076] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8952.636718] Lustre: 506688:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8952.665603] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8954.486301] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8957.896596] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8958.966319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8958.969414] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8958.997892] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8964.066330] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8964.066950] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8964.070455] LustreError: Skipped 1 previous similar message [ 8964.070865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8964.076196] Lustre: Skipped 1 previous similar message [ 8966.029034] Lustre: server umount lustre-OST0000 complete [ 8969.185874] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8969.186361] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8969.193608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8969.195271] Lustre: Skipped 2 previous similar messages [ 8973.404172] Lustre: server umount lustre-MDT0000 complete [ 8974.605147] LustreError: 506687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786488529 with bad export cookie 6606321567500781865 [ 8974.608218] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8974.610366] LustreError: 506687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8974.730505] Lustre: server umount lustre-MDT0001 complete [ 8978.337543] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 8979.299229] Key type lgssc unregistered [ 8979.437385] LNet: 508596:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8979.439918] LNetError: 508596:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8979.450289] LNet: Removed LNI 192.168.203.122@tcp [ 8979.748273] Key type .llcrypt unregistered [ 8979.749378] Key type ._llcrypt unregistered [ 8987.291825] Key type ._llcrypt registered [ 8987.292927] Key type .llcrypt registered [ 8987.325376] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 8987.631145] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8987.636827] alg: No test for adler32 (adler32-zlib) [ 8988.482955] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 8988.565816] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 8990.152233] Key type lgssc registered [ 8990.508183] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8993.033528] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8993.041173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8994.193780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8995.519937] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8998.369043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8998.497935] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8999.824129] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9002.407707] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9004.373232] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9006.731752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9006.872893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9008.922654] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9011.863575] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9026.486678] loop0: detected capacity change from 0 to 204800000 [ 9026.617151] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9026.626356] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9026.634152] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9029.308994] loop0: detected capacity change from 0 to 204800000 [ 9029.365028] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9029.369661] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9029.374372] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9031.506331] loop0: detected capacity change from 0 to 204800000 [ 9033.580144] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9033.652508] Lustre: fs15246-OST0000: new disk, initializing [ 9033.654427] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 9033.657243] Lustre: 513455:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9033.677791] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9036.063886] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9040.889145] loop1: detected capacity change from 0 to 204800000 [ 9043.212802] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9043.279987] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 9043.325419] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 9043.375549] Lustre: fs15246-MDT0000: new disk, initializing [ 9043.447620] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9043.459687] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9043.515295] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 9044.577053] Lustre: 514575:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000059e4c8b1 x1873269010498432/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786488615 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 9045.147202] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9053.670342] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 9053.678841] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 9053.693411] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 9056.231274] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9056.241502] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 9061.348425] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 9061.930102] Lustre: server umount fs15246-MDT0000 complete [ 9065.708828] Lustre: server umount fs15246-OST0000 complete [ 9070.561929] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9070.566255] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9070.573437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9071.590142] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9076.492422] Lustre: server umount lustre-OST0000 complete [ 9080.288796] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9080.291145] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9080.296677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9080.300500] Lustre: Skipped 2 previous similar messages [ 9084.003783] Lustre: server umount lustre-MDT0000 complete [ 9085.420709] LustreError: 509822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786488639 with bad export cookie 8759474803329918089 [ 9085.425026] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9085.425718] LustreError: 509822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9085.549698] Lustre: server umount lustre-MDT0001 complete [ 9089.446521] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 9090.456142] Key type lgssc unregistered [ 9090.599322] LNet: 517674:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9090.601826] LNetError: 517674:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9090.608326] LNet: Removed LNI 192.168.203.122@tcp [ 9090.941290] Key type .llcrypt unregistered [ 9090.942512] Key type ._llcrypt unregistered [ 9099.045147] Key type ._llcrypt registered [ 9099.046354] Key type .llcrypt registered [ 9099.085811] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 9099.519696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9099.558768] alg: No test for adler32 (adler32-zlib) [ 9100.429371] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 9100.543493] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 9102.136148] Key type lgssc registered [ 9102.525179] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9115.269219] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 9118.306674] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 9130.043430] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9130.050170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9131.146281] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9131.158402] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9131.205257] Lustre: lustre-MDT0000: new disk, initializing [ 9131.227953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9131.234119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9132.567409] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9136.783685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9136.834620] Lustre: 522108:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9136.861608] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9136.863835] Lustre: Skipped 1 previous similar message [ 9136.906141] Lustre: lustre-MDT0001: new disk, initializing [ 9136.939381] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9136.954552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9136.958336] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9138.254916] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9141.930113] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9143.717504] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9146.126098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9146.236457] Lustre: lustre-OST0000: new disk, initializing [ 9146.238991] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9146.242572] Lustre: 524076:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9146.276307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9148.272493] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9151.996656] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9153.532113] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9155.575845] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9155.578979] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9155.610032] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9156.672304] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 9158.011661] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9158.080056] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9160.675612] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9160.676515] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9160.679982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9160.681030] LustreError: Skipped 1 previous similar message [ 9160.690543] Lustre: Skipped 1 previous similar message [ 9164.839632] Lustre: server umount lustre-OST0000 complete [ 9167.841601] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9167.846965] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9167.856212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9167.860809] Lustre: Skipped 1 previous similar message [ 9170.914099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9170.914411] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9172.265904] Lustre: server umount lustre-MDT0000 complete [ 9173.901171] LustreError: 522103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786488728 with bad export cookie 4279669471219066774 [ 9173.904087] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9173.906109] LustreError: 522103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9174.011133] Lustre: server umount lustre-MDT0001 complete [ 9184.517164] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 9188.824625] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 9200.550550] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 9039 sec ========== 18:52:34 (1786488754) [ 9201.103927] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 18:52:35 (1786488755) === [ 9202.266650] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 18:52:36 (1786488756) === [ 9217.381482] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 9218.564266] Key type lgssc unregistered [ 9218.713316] LNet: 530732:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9218.715759] LNetError: 530732:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9218.724204] LNet: Removed LNI 192.168.203.122@tcp [ 9219.059666] Key type .llcrypt unregistered [ 9219.061341] Key type ._llcrypt unregistered