[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 816790203 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002000] x2apic enabled [ 0.002009] Switched APIC routing to physical x2apic. [ 0.003021] kvm-guest: setup PV IPIs [ 0.006627] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008010] pid_max: default: 32768 minimum: 301 [ 0.009128] LSM: Security Framework initializing [ 0.010062] Yama: becoming mindful. [ 0.011040] SELinux: Initializing. [ 0.012056] *** VALIDATE selinux *** [ 0.022165] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027417] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029010] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030119] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031132] *** VALIDATE tmpfs *** [ 0.032447] *** VALIDATE proc *** [ 0.034257] *** VALIDATE cgroup *** [ 0.035008] *** VALIDATE cgroup2 *** [ 0.036251] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037153] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039034] Spectre V2 : User space: Vulnerable [ 0.040008] Speculative Store Bypass: Vulnerable [ 0.043638] debug: unmapping init [mem 0xffffffffb9a59000-0xffffffffb9a60fff] [ 0.046000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046735] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047023] ... version: 2 [ 0.048011] ... bit width: 48 [ 0.049011] ... generic registers: 4 [ 0.050011] ... value mask: 0000ffffffffffff [ 0.051017] ... max period: 00007fffffffffff [ 0.052012] ... fixed-purpose events: 3 [ 0.053009] ... event mask: 000000070000000f [ 0.054309] rcu: Hierarchical SRCU implementation. [ 0.056433] smp: Bringing up secondary CPUs ... [ 0.057554] x86: Booting SMP configuration: [ 0.058021] .... node #0, CPUs: #1 #2 #3 [ 0.071273] smp: Brought up 1 node, 4 CPUs [ 0.073016] smpboot: Max logical packages: 1 [ 0.074012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.112190] node 0 deferred pages initialised in 36ms [ 0.117253] devtmpfs: initialized [ 0.118281] x86/mm: Memory block size: 128MB [ 0.122651] gcov: version magic: 0x41383552 [ 0.125514] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.126157] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.128160] pinctrl core: initialized pinctrl subsystem [ 0.129250] [ 0.130008] ************************************************************* [ 0.131013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.132012] ** ** [ 0.133016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.134012] ** ** [ 0.135011] ** This means that this kernel is built to expose internal ** [ 0.136013] ** IOMMU data structures, which may compromise security on ** [ 0.137013] ** your system. ** [ 0.138012] ** ** [ 0.139012] ** If you see this message and you are not debugging the ** [ 0.140012] ** kernel, report this immediately to your vendor! ** [ 0.141123] ** ** [ 0.142018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.143012] ************************************************************* [ 0.144773] NET: Registered protocol family 16 [ 0.145526] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.146059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.147092] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.148685] cpuidle: using governor menu [ 0.149916] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.150520] PCI: Using configuration type 1 for base access [ 0.151247] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.159175] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.160000] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.162223] cryptd: max_cpu_qlen set to 1000 [ 0.164097] ACPI: Added _OSI(Module Device) [ 0.166014] ACPI: Added _OSI(Processor Device) [ 0.168011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.169011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.174495] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.182542] ACPI: Interpreter enabled [ 0.183058] ACPI: PM: (supports S0 S3 S4 S5) [ 0.184016] ACPI: Using IOAPIC for interrupt routing [ 0.185097] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.186407] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.195919] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.198034] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.200017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.204078] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.211332] acpiphp: Slot [2] registered [ 0.213143] acpiphp: Slot [5] registered [ 0.214107] acpiphp: Slot [6] registered [ 0.215175] acpiphp: Slot [7] registered [ 0.217089] acpiphp: Slot [8] registered [ 0.218074] acpiphp: Slot [9] registered [ 0.219094] acpiphp: Slot [10] registered [ 0.221076] acpiphp: Slot [3] registered [ 0.222148] acpiphp: Slot [4] registered [ 0.223066] acpiphp: Slot [11] registered [ 0.225118] acpiphp: Slot [12] registered [ 0.226078] acpiphp: Slot [13] registered [ 0.227086] acpiphp: Slot [14] registered [ 0.228224] acpiphp: Slot [15] registered [ 0.230119] acpiphp: Slot [16] registered [ 0.232094] acpiphp: Slot [17] registered [ 0.234090] acpiphp: Slot [18] registered [ 0.235096] acpiphp: Slot [19] registered [ 0.238227] acpiphp: Slot [20] registered [ 0.240108] acpiphp: Slot [21] registered [ 0.242280] acpiphp: Slot [22] registered [ 0.245437] acpiphp: Slot [23] registered [ 0.248099] acpiphp: Slot [24] registered [ 0.250220] acpiphp: Slot [25] registered [ 0.253109] acpiphp: Slot [26] registered [ 0.255254] acpiphp: Slot [27] registered [ 0.258113] acpiphp: Slot [28] registered [ 0.260109] acpiphp: Slot [29] registered [ 0.262300] acpiphp: Slot [30] registered [ 0.264108] acpiphp: Slot [31] registered [ 0.266059] PCI host bridge to bus 0000:00 [ 0.268018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.271023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.275023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.279022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.282036] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.287030] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.290281] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.293086] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.298406] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.315014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.327550] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.341020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.347017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.356018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.362675] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.365748] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.367036] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.370860] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.376015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.390014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.395012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.400339] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.418023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.434017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.465017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.478491] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.492025] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.503018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.528033] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.551052] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.592017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.603048] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.638015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.662505] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.690014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.698015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.720015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.733614] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.742017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.749016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.769017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.779000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.787017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.795027] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.818062] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.838000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.842431] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.845723] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.848429] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.851418] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.860290] iommu: Default domain type: Passthrough [ 0.865012] SCSI subsystem initialized [ 0.867158] ACPI: bus type USB registered [ 0.869134] usbcore: registered new interface driver usbfs [ 0.871190] usbcore: registered new interface driver hub [ 0.874126] usbcore: registered new device driver usb [ 0.877274] pps_core: LinuxPPS API ver. 1 registered [ 0.879011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.883171] PTP clock support registered [ 0.886293] EDAC MC: Ver: 3.0.0 [ 0.889172] PCI: Using ACPI for IRQ routing [ 0.898624] NetLabel: Initializing [ 0.901011] NetLabel: domain hash size = 128 [ 0.903012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.907079] NetLabel: unlabeled traffic allowed by default [ 0.909481] vgaarb: loaded [ 0.912141] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.916014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.921262] clocksource: Switched to clocksource kvm-clock [ 1.118647] VFS: Disk quotas dquot_6.6.0 [ 1.120088] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.122787] *** VALIDATE ramfs *** [ 1.124031] *** VALIDATE hugetlbfs *** [ 1.125630] pnp: PnP ACPI init [ 1.127966] pnp: PnP ACPI: found 6 devices [ 1.143886] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.147069] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.149288] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.151474] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.153936] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.156516] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.159510] NET: Registered protocol family 2 [ 1.162024] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.166912] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.170528] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.175707] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.179308] TCP: Hash tables configured (established 65536 bind 65536) [ 1.182223] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.185359] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.188399] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.191470] NET: Registered protocol family 1 [ 1.194122] RPC: Registered named UNIX socket transport module. [ 1.196053] RPC: Registered udp transport module. [ 1.197781] RPC: Registered tcp transport module. [ 1.199433] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.201769] NET: Registered protocol family 44 [ 1.203353] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.205480] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.207558] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.209848] PCI: CLS 0 bytes, default 64 [ 1.211456] Unpacking initramfs... [ 3.084129] debug: unmapping init [mem 0xffff97287cc54000-0xffff97287ffbffff] [ 3.089920] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.092410] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.096274] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.843867] Initialise system trusted keyrings [ 3.845544] Key type blacklist registered [ 3.847549] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.856720] zbud: loaded [ 3.859753] *** VALIDATE nfs *** [ 3.860941] *** VALIDATE nfs4 *** [ 3.862535] pstore: using deflate compression [ 3.866468] Platform Keyring initialized [ 4.122431] NET: Registered protocol family 38 [ 4.126509] Key type asymmetric registered [ 4.128308] Asymmetric key parser 'x509' registered [ 4.130359] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.133621] io scheduler mq-deadline registered [ 4.135382] io scheduler kyber registered [ 4.137253] io scheduler bfq registered [ 4.139311] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.142555] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.146677] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.149927] ACPI: Power Button [PWRF] [ 4.155777] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.162669] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.189277] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.201230] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.232970] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.263867] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.302915] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.307672] Non-volatile memory driver v1.3 [ 4.309311] Linux agpgart interface v0.103 [ 4.355331] virtio_blk virtio1: [vda] 133608 512-byte logical blocks (68.4 MB/65.2 MiB) [ 4.359849] vda: detected capacity change from 0 to 68407296 [ 4.392982] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.396678] vdb: detected capacity change from 0 to 1073741824 [ 4.417677] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.420957] vdc: detected capacity change from 0 to 2621440000 [ 4.451201] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.454460] vdd: detected capacity change from 0 to 2621440000 [ 4.483783] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.488908] vde: detected capacity change from 0 to 4294967296 [ 4.534475] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.539799] vdf: detected capacity change from 0 to 4294967296 [ 4.558910] libphy: Fixed MDIO Bus: probed [ 4.569556] usbcore: registered new interface driver usbserial_generic [ 4.579146] usbserial: USB Serial support registered for generic [ 4.617789] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.624617] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.627801] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.630992] mousedev: PS/2 mouse device common for all mice [ 4.636326] rtc_cmos 00:05: RTC can wake from S4 [ 4.639352] rtc_cmos 00:05: registered as rtc0 [ 4.641347] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.645074] intel_pstate: CPU model not supported [ 4.648277] hid: raw HID events driver (C) Jiri Kosina [ 4.650318] usbcore: registered new interface driver usbhid [ 4.652589] usbhid: USB HID core driver [ 4.654392] drop_monitor: Initializing network drop monitor service [ 4.657132] Initializing XFRM netlink socket [ 4.659158] NET: Registered protocol family 10 [ 4.660697] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.665302] Segment Routing with IPv6 [ 4.665339] NET: Registered protocol family 17 [ 4.665444] mpls_gso: MPLS GSO support [ 4.678591] RAS: Correctable Errors collector initialized. [ 4.680620] AVX version of gcm_enc/dec engaged. [ 4.682923] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.686552] AES CTR mode by8 optimization enabled [ 4.703137] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.786393] sched_clock: Marking stable (4786320241, 0)->(6299587171, -1513266930) [ 4.792404] registered taskstats version 1 [ 4.794187] Loading compiled-in X.509 certificates [ 4.796406] zswap: loaded using pool lzo/zbud [ 4.821845] Key type big_key registered [ 4.833340] Key type encrypted registered [ 4.834462] ima: No TPM chip found, activating TPM-bypass! [ 4.836391] ima: Allocated hash algorithm: sha1 [ 4.838213] ima: No architecture policies found [ 4.839468] evm: Initialising EVM extended attributes: [ 4.840795] evm: security.selinux [ 4.841751] evm: security.ima [ 4.842594] evm: security.capability [ 4.843549] evm: HMAC attrs: 0x1 [ 4.845574] rtc_cmos 00:05: setting system clock to 2026-03-11 13:14:22 UTC (1773234862) [ 4.852857] debug: unmapping init [mem 0xffffffffbaa03000-0xffffffffbabfffff] [ 4.855425] debug: unmapping init [mem 0xffffffffb9782000-0xffffffffb9a58fff] [ 4.866123] Write protecting the kernel read-only data: 28672k [ 4.869377] debug: unmapping init [mem 0xffffffffb7e03000-0xffffffffb7ffffff] [ 4.871633] debug: unmapping init [mem 0xffffffffb8714000-0xffffffffb87fffff] [ 4.903426] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.909750] systemd[1]: Detected virtualization kvm. [ 4.911397] systemd[1]: Detected architecture x86-64. [ 4.916449] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.949556] systemd[1]: No hostname configured. [ 4.952542] systemd[1]: Set hostname to . [ 4.955557] random: systemd: uninitialized urandom read (16 bytes read) [ 4.957857] systemd[1]: Initializing machine ID from random generator. [ 5.034400] random: ln: uninitialized urandom read (6 bytes read) [ 5.418646] random: systemd: uninitialized urandom read (16 bytes read) [ 5.425343] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 5.431275] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 5.446806] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. [ OK ] Listening on udev Kernel Socket. Starting Create Volatile Files and Directories... [ OK ] Reached target Timers. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on Journal Socket (/dev/log). Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... Starting Setup Virtual Console... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.763988] device-mapper: uevent: version 1.0.3 [ 6.766257] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 8.622956] random: fast init done [ 8.830327] virtio_net virtio0 ens2: renamed from eth0 [ 10.340983] scsi host0: ata_piix [ 10.734901] scsi host1: ata_piix [ 10.741867] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 10.748667] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 15.385218] random: crng init done [ 15.386913] random: 7 urandom warning(s) missed due to ratelimiting [ 18.457620] dracut-initqueue[592]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 20.145310] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 23.350752] printk: systemd: 26 output lines suppressed due to ratelimiting [ 24.167187] SELinux: Disabled at runtime. [ 24.243308] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 24.257495] systemd[1]: Detected virtualization kvm. [ 24.259668] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 25.813830] systemd[1]: initrd-switch-root.service: Succeeded. [ 25.824895] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 25.838245] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 25.858830] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 25.863755] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 25.874833] systemd[1]: Starting Journal Service... Starting Journal Service... [ 25.886647] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Initrd Root File System. Mounting Huge Pages File System... Starting Apply Kernel Variables... Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ 26.633443] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 28.075918] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 29.451979] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 30.340906] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 32.329902] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 32.561928] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (11s / no limit) [ *] A start job is running for Configur…only root support (12s / no limit) [ **] A start job is running for Configur…only root support (13s / no limit) [ ***] A start job is running for Configur…only root support (13s / no limit) [ *** ] A start job is running for Configur…only root support (14s / no limit)[ 40.327221] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (14s / no limit) [*** ] A start job is running for Configur…only root support (15s / no limit)[ 41.283901] NFS: Registering the id_resolver key type [ 41.286494] Key type id_resolver registered [ 41.288265] Key type id_legacy registered [** ] A start job is running for Configur…only root support (15s / no limit) [* ] A start job is running for Configur…only root support (16s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Started Login Service. Starting Hostname Service... Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg311-server login: [ 84.056742] libcfs: loading out-of-tree module taints kernel. [ 84.083781] Key type ._llcrypt registered [ 84.085592] Key type .llcrypt registered [ 84.150452] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_hostid [ 94.664873] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 95.568260] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 95.576682] alg: No test for adler32 (adler32-zlib) [ 96.745418] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 97.244122] LNet: Added LNI 192.168.203.111@tcp [8/256/0/180] [ 98.888193] Key type lgssc registered [ 99.706057] Lustre: Echo OBD driver; http://www.lustre.org/ [ 110.113746] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 110.897772] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 114.663183] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 118.349985] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 125.726844] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 133.962795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 133.992931] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 134.013189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 135.189043] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 135.219634] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 135.276859] Lustre: lustre-MDT0000: new disk, initializing [ 135.353294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 135.367726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 138.180692] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 141.528219] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 147.563495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 147.608073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 147.766675] Lustre: lustre-OST0000: new disk, initializing [ 147.775074] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 147.777111] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 147.778059] Lustre: Skipped 1 previous similar message [ 147.811140] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 151.499323] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 153.085985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 153.091757] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 153.111226] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 160.553907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 160.623605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 160.673367] Lustre: lustre-OST0001: new disk, initializing [ 160.676599] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 160.700088] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 163.312455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 163.315421] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 163.333054] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 164.141859] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 173.156271] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 180.952581] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 189.464209] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing check_logdir /tmp/testlogs/ [ 191.761956] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing yml_node [ 193.559476] Lustre: DEBUG MARKER: Client: 2.16.59.50 [ 194.590350] Lustre: DEBUG MARKER: MDS: 2.16.59.50 [ 195.594217] Lustre: DEBUG MARKER: OSS: 2.16.59.50 [ 196.221558] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Mar 11 09:17:33 EDT 2026 [ 207.130971] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 208.085153] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 224.738699] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 224.741036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 224.753087] Lustre: Skipped 1 previous similar message [ 224.764461] Lustre: Skipped 1 previous similar message [ 227.170630] LustreError: 10692:0:(obd_class.h:479:obd_check_dev()) Device 17 not setup [ 227.278045] Lustre: server umount lustre-MDT0000 complete [ 229.830582] LustreError: 8758:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773235087 with bad export cookie 16091881072340170997 [ 229.831348] LustreError: MGC192.168.203.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 229.837738] LustreError: 8758:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 229.874111] LustreError: 10893:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 229.881874] LustreError: 10893:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 229.912460] Lustre: server umount lustre-OST0000 complete [ 232.100108] LustreError: 11094:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 232.103140] LustreError: 11094:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 232.170224] Lustre: server umount lustre-OST0001 complete [ 235.986958] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_hostid [ 241.221976] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 247.708683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 251.188412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 255.380820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 260.077785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 260.134192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 260.341630] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 260.372342] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 260.422581] Lustre: lustre-MDT0000: new disk, initializing [ 260.494390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 260.507352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 262.851803] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 268.261560] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 273.046743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 273.083911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 273.222798] Lustre: lustre-OST0000: new disk, initializing [ 273.226528] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 273.231375] Lustre: Skipped 1 previous similar message [ 273.274810] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 274.802813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 274.815702] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 274.838801] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 277.501293] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 282.897633] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 285.833852] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 285.975412] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 290.276379] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 290.282160] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 290.291781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 293.242248] LustreError: 14831:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 293.245339] LustreError: 14831:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 293.374699] Lustre: server umount lustre-OST0000 complete [ 296.315920] Lustre: server umount lustre-MDT0000 complete [ 299.110486] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 09:19:15 (1773235155) [ 300.825530] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 301.929864] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 09:19:18 (1773235158) [ 306.482840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 306.830191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 309.908678] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 313.691627] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 318.800237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 323.202117] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 326.656325] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 342.498068] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 342.511880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 347.575140] LustreError: 17172:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 347.578749] LustreError: 17172:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 347.626720] Lustre: server umount lustre-OST0000 complete [ 350.076564] Lustre: server umount lustre-MDT0000 complete [ 355.992743] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing unload_modules_local [ 357.842877] Key type lgssc unregistered [ 358.039852] LNet: 17905:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 358.044188] LNetError: 17905:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 358.055659] LNet: Removed LNI 192.168.203.111@tcp [ 358.626115] Key type .llcrypt unregistered [ 358.627077] Key type ._llcrypt unregistered [ 372.415571] Key type ._llcrypt registered [ 372.417291] Key type .llcrypt registered [ 372.484547] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 373.186984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 373.262080] alg: No test for adler32 (adler32-zlib) [ 374.211330] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 374.371121] LNet: Added LNI 192.168.203.111@tcp [8/256/0/180] [ 376.016458] Key type lgssc registered [ 376.686836] Lustre: Echo OBD driver; http://www.lustre.org/ [ 381.804422] loop: module loaded [ 381.857204] loop0: detected capacity change from 0 to 204800000 [ 381.998950] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 382.010682] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 382.260387] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 382.309841] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 386.644291] loop0: detected capacity change from 0 to 204800000 [ 386.819178] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 386.827426] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 386.835083] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 386.990819] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 390.070235] loop0: detected capacity change from 0 to 204800000 [ 393.143518] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 393.172377] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 393.181167] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 394.317745] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 394.353730] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 394.410700] Lustre: lustre-MDT0000: new disk, initializing [ 394.471814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 394.481979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 394.493420] Lustre: 20016:0:(osd_internal.h:1471:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 466, rollback = 2 [ 394.497515] Lustre: 20016:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 394.499509] Lustre: 20016:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 2/466/0 [ 394.502316] Lustre: 20016:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 394.505442] Lustre: 20016:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0 [ 394.507799] Lustre: 20016:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 396.975671] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 400.700608] hrtimer: interrupt took 4482355 ns [ 402.786830] loop1: detected capacity change from 0 to 204800000 [ 406.688800] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 406.727138] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 406.911817] Lustre: lustre-OST07c6: new disk, initializing [ 406.914216] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 406.916390] Lustre: Skipped 1 previous similar message [ 406.974874] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 409.296504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 409.304568] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 409.337335] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 410.885829] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 415.610058] Lustre: 21223:0:(osd_internal.h:1471:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 415.615315] Lustre: 21223:0:(osd_internal.h:1471:osd_trans_exec_op()) Skipped 1 previous similar message [ 415.619447] Lustre: 21223:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 415.624253] Lustre: 21223:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 1 previous similar message [ 415.629403] Lustre: 21223:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 415.636185] Lustre: 21223:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 415.641235] Lustre: 21223:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 415.645310] Lustre: 21223:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 1 previous similar message [ 415.652000] Lustre: 21223:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 415.654730] Lustre: 21223:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 415.661607] Lustre: 21223:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 415.667141] Lustre: 21223:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 436.211985] Lustre: 20040:0:(osd_internal.h:1471:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 497, rollback = 2 [ 436.219248] Lustre: 20040:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 436.224427] Lustre: 20040:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/497/0 [ 436.228064] Lustre: 20040:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 436.233676] Lustre: 20040:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 436.238094] Lustre: 20040:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 440.409974] Lustre: 20040:0:(osd_internal.h:1471:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 497, rollback = 2 [ 440.415100] Lustre: 20040:0:(osd_internal.h:1471:osd_trans_exec_op()) Skipped 1 previous similar message [ 440.418959] Lustre: 20040:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 440.422370] Lustre: 20040:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 1 previous similar message [ 440.426066] Lustre: 20040:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/497/0 [ 440.428628] Lustre: 20040:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 440.431800] Lustre: 20040:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 440.437774] Lustre: 20040:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 1 previous similar message [ 440.444131] Lustre: 20040:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 440.450433] Lustre: 20040:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 440.454577] Lustre: 20040:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 440.457768] Lustre: 20040:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 447.963314] Lustre: 20040:0:(osd_internal.h:1471:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 497, rollback = 2 [ 447.968368] Lustre: 20040:0:(osd_internal.h:1471:osd_trans_exec_op()) Skipped 3 previous similar messages [ 447.974562] Lustre: 20040:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 447.979193] Lustre: 20040:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 447.983291] Lustre: 20040:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/497/0 [ 447.989391] Lustre: 20040:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 447.994134] Lustre: 20040:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 447.997720] Lustre: 20040:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 448.001933] Lustre: 20040:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 448.004698] Lustre: 20040:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 448.008311] Lustre: 20040:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 448.012393] Lustre: 20040:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 459.485262] Lustre: 21223:0:(osd_internal.h:1471:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 497, rollback = 2 [ 459.494947] Lustre: 21223:0:(osd_internal.h:1471:osd_trans_exec_op()) Skipped 5 previous similar messages [ 459.501460] Lustre: 21223:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 459.505020] Lustre: 21223:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 459.508736] Lustre: 21223:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/497/0 [ 459.512149] Lustre: 21223:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 459.515555] Lustre: 21223:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 459.518846] Lustre: 21223:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 459.522351] Lustre: 21223:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 459.525744] Lustre: 21223:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 459.528928] Lustre: 21223:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 459.532513] Lustre: 21223:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 506.850799] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 506.853149] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 506.859051] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 511.969505] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 512.494949] LustreError: 22229:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 512.619327] Lustre: server umount lustre-OST07c6 complete [ 518.816446] LustreError: 22730:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 518.819319] LustreError: 22730:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 518.940502] Lustre: server umount lustre-MDT0000 complete [ 528.929996] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing unload_modules_local [ 530.621402] Key type lgssc unregistered [ 530.820403] LNet: 23561:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 530.823854] LNetError: 23561:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 531.878760] LNet: Removed LNI 192.168.203.111@tcp [ 532.385240] Key type .llcrypt unregistered [ 532.387200] Key type ._llcrypt unregistered [ 542.788181] Key type ._llcrypt registered [ 542.790035] Key type .llcrypt registered [ 542.875398] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 543.644944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 543.657181] alg: No test for adler32 (adler32-zlib) [ 544.574926] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 544.708496] LNet: Added LNI 192.168.203.111@tcp [8/256/0/180] [ 546.312144] Key type lgssc registered [ 546.954275] Lustre: Echo OBD driver; http://www.lustre.org/ [ 551.919282] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 551.955941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 553.348197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 556.067759] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 559.799236] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 563.940865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 564.201749] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 567.478362] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 570.586790] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 577.452914] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:35 to 0x240000400:65) [ 585.029880] LustreError: 26208:0:(qsd_reint.c:618:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:2 until pending updates are flushed. [ 590.317096] LustreError: 26455:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 590.376605] Lustre: server umount lustre-OST0000 complete [ 603.114253] LustreError: 26655:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 603.116916] LustreError: 26655:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 603.226406] Lustre: server umount lustre-MDT0000 complete [ 609.671306] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing unload_modules_local [ 611.327780] Key type lgssc unregistered [ 611.515693] LNet: 27189:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 611.520391] LNetError: 27189:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 612.588233] LNet: Removed LNI 192.168.203.111@tcp [ 613.128849] Key type .llcrypt unregistered [ 613.130913] Key type ._llcrypt unregistered [ 624.064933] Key type ._llcrypt registered [ 624.066739] Key type .llcrypt registered [ 624.135651] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 624.904668] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 624.927198] alg: No test for adler32 (adler32-zlib) [ 625.931622] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 626.096616] LNet: Added LNI 192.168.203.111@tcp [8/256/0/180] [ 627.760171] Key type lgssc registered [ 628.478608] Lustre: Echo OBD driver; http://www.lustre.org/ [ 633.193236] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 633.205035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 634.488486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 637.105242] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 640.352582] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 644.575463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 644.777687] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 645.814805] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:67 to 0x240000400:97) [ 648.775507] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 652.337895] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 661.380139] LustreError: 29868:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 661.452929] Lustre: server umount lustre-OST0000 complete [ 674.281340] LustreError: 30069:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 674.287204] LustreError: 30069:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 674.375455] Lustre: server umount lustre-MDT0000 complete [ 680.881543] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing unload_modules_local [ 682.395926] Key type lgssc unregistered [ 682.569938] LNet: 30601:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 682.581950] LNetError: 30601:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 682.602755] LNet: Removed LNI 192.168.203.111@tcp [ 683.030630] Key type .llcrypt unregistered [ 683.033412] Key type ._llcrypt unregistered [ 695.688863] Key type ._llcrypt registered [ 695.690505] Key type .llcrypt registered [ 695.740409] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 696.181670] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 696.196201] alg: No test for adler32 (adler32-zlib) [ 697.126269] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 697.263526] LNet: Added LNI 192.168.203.111@tcp [8/256/0/180] [ 698.872719] Key type lgssc registered [ 699.486640] Lustre: Echo OBD driver; http://www.lustre.org/ [ 703.306824] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 703.319789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 704.599600] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 706.729585] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 709.788842] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 713.835103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 714.018875] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 715.046345] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 717.007845] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 719.808149] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 725.476289] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 725.486062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 726.271294] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.11@tcp (stopping) [ 729.941710] LustreError: 33419:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 730.041309] Lustre: server umount lustre-MDT0000 complete [ 741.500172] LustreError: 31956:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.11@tcp arrived at 1773235599 with bad export cookie 1771746270576250508 [ 741.513304] LustreError: 31956:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 746.976297] Lustre: 31285:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773235588/real 1773235588] req@000000008dc39ded x1859372244280704/t0(0) o400->MGC192.168.203.111@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1773235604 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 746.994336] LustreError: MGC192.168.203.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 759.416240] LustreError: 33645:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 759.419237] LustreError: 33645:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 759.523200] Lustre: server umount lustre-OST0000 complete [ 767.619380] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing unload_modules_local [ 769.430621] Key type lgssc unregistered [ 769.610596] LNet: 34330:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 769.612922] LNetError: 34330:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 769.621471] LNet: Removed LNI 192.168.203.111@tcp [ 770.102252] Key type .llcrypt unregistered [ 770.104266] Key type ._llcrypt unregistered [ 783.458961] Key type ._llcrypt registered [ 783.462422] Key type .llcrypt registered [ 783.539812] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 784.151829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 784.248059] alg: No test for adler32 (adler32-zlib) [ 785.194803] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 785.326919] LNet: Added LNI 192.168.203.111@tcp [8/256/0/180] [ 786.976259] Key type lgssc registered [ 787.726505] Lustre: Echo OBD driver; http://www.lustre.org/ [ 792.659302] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 792.673688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 794.011497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 796.563600] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 800.026326] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 804.143348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 804.352755] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 808.031652] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 808.253195] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:131 to 0x240000400:161) [ 811.137790] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 818.661589] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 818.667418] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 818.676791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 819.681136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 819.684375] Lustre: Skipped 1 previous similar message [ 822.580232] LustreError: 37150:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 822.638271] Lustre: server umount lustre-OST0000 complete [ 843.859971] LustreError: 37525:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 843.862679] LustreError: 37525:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 843.974558] Lustre: server umount lustre-MDT0000 complete [ 851.035244] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing unload_modules_local [ 852.976020] Key type lgssc unregistered [ 853.183280] LNet: 38058:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 853.189484] LNetError: 38058:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 853.199222] LNet: Removed LNI 192.168.203.111@tcp [ 853.699189] Key type .llcrypt unregistered [ 853.704609] Key type ._llcrypt unregistered [ 866.586493] Key type ._llcrypt registered [ 866.587520] Key type .llcrypt registered [ 866.639631] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 867.205465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 867.275260] alg: No test for adler32 (adler32-zlib) [ 868.200695] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 868.314419] LNet: Added LNI 192.168.203.111@tcp [8/256/0/180] [ 869.928184] Key type lgssc registered [ 870.524738] Lustre: Echo OBD driver; http://www.lustre.org/ [ 874.632243] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 874.641581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 875.874931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 877.786990] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 880.620271] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 883.990020] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 884.168342] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 886.257637] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:163 to 0x240000400:193) [ 887.578644] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 890.747432] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 895.336111] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 897.087526] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 908.207472] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-03-11 9h29m24s [ 909.005246] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 911.845987] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 911.856663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 912.932457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 914.665318] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.11@tcp (stopping) [ 914.668269] Lustre: Skipped 1 previous similar message [ 916.304894] LustreError: 41196:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 916.429635] Lustre: server umount lustre-MDT0000 complete [ 918.457682] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 922.041781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 922.122744] LustreError: MGC192.168.203.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 922.342925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 922.395424] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:163 to 0x240000400:225) [ 924.604170] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 926.487415] Lustre: DEBUG MARKER: Wait for df (30872) ... [ 927.378559] Lustre: DEBUG MARKER: done [ 929.769047] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 929.779258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 934.773510] LustreError: 42258:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 934.776584] LustreError: 42258:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 934.836286] Lustre: server umount lustre-OST0000 complete [ 936.918811] LustreError: 42459:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 936.923121] LustreError: 42459:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 937.048978] Lustre: server umount lustre-MDT0000 complete [ 942.147431] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing unload_modules_local [ 943.455096] Key type lgssc unregistered [ 943.634830] LNet: 42991:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 943.638093] LNetError: 42991:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 943.649420] LNet: Removed LNI 192.168.203.111@tcp [ 944.044941] Key type .llcrypt unregistered [ 944.046434] Key type ._llcrypt unregistered [ 956.511750] Key type ._llcrypt registered [ 956.514114] Key type .llcrypt registered [ 956.582115] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_hostid [ 964.617326] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 965.264052] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 965.274577] alg: No test for adler32 (adler32-zlib) [ 966.199943] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 966.315774] LNet: Added LNI 192.168.203.111@tcp [8/256/0/180] [ 967.928160] Key type lgssc registered [ 968.518606] Lustre: Echo OBD driver; http://www.lustre.org/ [ 973.257327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 976.758434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 979.997424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 983.663777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 983.705804] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 983.722749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 984.852671] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 984.870440] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 984.912987] Lustre: lustre-MDT0000: new disk, initializing [ 984.984773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 985.000926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 986.850191] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 991.037179] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 994.657463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 994.699581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 994.837633] Lustre: lustre-OST0000: new disk, initializing [ 994.840958] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 994.843805] Lustre: Skipped 1 previous similar message [ 994.883174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 998.047628] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 998.742338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 998.747233] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 998.762230] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1002.704989] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1005.037194] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1005.141280] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1009.123733] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1009.124137] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1009.132607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1012.340124] LustreError: 48122:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 1012.408611] Lustre: server umount lustre-OST0000 complete [ 1014.321704] LustreError: 48323:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 1014.327174] LustreError: 48323:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1014.413562] Lustre: server umount lustre-MDT0000 complete [ 1018.306505] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 09:31:15 (1773235875) [ 1021.216472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1021.459809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1023.089535] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 1025.478402] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1028.342258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1028.501864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1031.054043] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 1033.860881] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1037.719366] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-03-11 9h31m34s [ 1038.429907] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1041.230081] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1042.235896] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1042.251670] LustreError: 48989:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1062.651317] Lustre: lustre-MDT0000: Client ecdb60c4-ce7f-41b4-9db6-359e35456613 (at 192.168.203.11@tcp) reconnecting [ 1082.296079] LustreError: 48989:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1083.131901] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1084.096433] Lustre: DEBUG MARKER: done [ 1087.969776] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1087.969908] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1087.982657] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1091.884113] LustreError: 51193:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 1091.886080] LustreError: 51193:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1091.917904] Lustre: server umount lustre-OST0000 complete [ 1093.359739] Lustre: server umount lustre-MDT0000 complete [ 1097.205838] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing unload_modules_local [ 1098.382605] Key type lgssc unregistered [ 1098.517418] LNet: 51926:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1098.520623] LNetError: 51926:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1098.534363] LNet: Removed LNI 192.168.203.111@tcp [ 1098.889160] Key type .llcrypt unregistered [ 1098.890827] Key type ._llcrypt unregistered [ 1108.505862] Key type ._llcrypt registered [ 1108.507114] Key type .llcrypt registered [ 1108.548815] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_hostid [ 1115.114323] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 1115.432819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1115.509352] alg: No test for adler32 (adler32-zlib) [ 1116.381531] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 1116.469188] LNet: Added LNI 192.168.203.111@tcp [8/256/0/180] [ 1118.048140] Key type lgssc registered [ 1118.430498] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1121.524317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1123.949589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1126.297738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1128.759696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1128.780607] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1128.787538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1129.875908] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1129.888869] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1129.928306] Lustre: lustre-MDT0000: new disk, initializing [ 1129.967361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1129.980070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1131.440671] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 1134.794794] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1137.409848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1137.439264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1137.544385] Lustre: lustre-OST0000: new disk, initializing [ 1137.547310] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1137.550973] Lustre: Skipped 1 previous similar message [ 1137.587326] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1139.955199] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 1143.503133] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1145.007832] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1145.349633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1145.355254] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1145.368819] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1146.113378] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 1150.433603] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1150.434246] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1150.442823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1152.880119] LustreError: 57067:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 1152.920546] Lustre: server umount lustre-OST0000 complete [ 1154.454632] LustreError: 57268:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 1154.457090] LustreError: 57268:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1154.526072] Lustre: server umount lustre-MDT0000 complete [ 1157.807573] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 09:33:34 (1773236014) [ 1158.406691] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1159.082954] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 09:33:36 (1773236016) [ 1163.142492] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 1164.733914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1164.932753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1165.351310] Lustre: Failing over lustre-MDT0000 [ 1165.418180] LustreError: 58392:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 1165.420298] LustreError: 58392:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1165.477966] Lustre: server umount lustre-MDT0000 complete [ 1168.408747] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 09:33:45 (1773236025) [ 1171.264981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1171.457917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1173.068968] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 1175.285069] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1177.820134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1177.938803] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1180.191040] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 1182.316920] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1185.504846] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1192.417376] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1192.422878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1193.603391] LustreError: 60505:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 1193.606376] LustreError: 60505:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1193.665488] Lustre: server umount lustre-MDT0000 complete [ 1194.955309] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1196.750516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1200.046475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1200.116856] LustreError: MGC192.168.203.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1200.290192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1200.337085] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 1201.813394] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 1203.968090] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1212.385378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1212.422815] LustreError: 61968:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 1212.425349] LustreError: 61968:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1212.478798] Lustre: server umount lustre-MDT0000 complete [ 1215.075739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1218.681063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1218.733864] LustreError: MGC192.168.203.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1218.874362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1218.894200] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 1220.366191] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 1222.487053] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1230.606938] LustreError: 63428:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 1230.610712] LustreError: 63428:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1230.685604] Lustre: server umount lustre-MDT0000 complete [ 1233.514077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1234.857374] Lustre: DEBUG MARKER: files compared the same [ 1235.664276] LustreError: 63342:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773236093 with bad export cookie 18246788915448193096 [ 1235.666905] LustreError: MGC192.168.203.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1235.669121] LustreError: 63342:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1235.733464] Lustre: server umount lustre-OST0000 complete [ 1240.895323] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing unload_modules_local [ 1242.098799] Key type lgssc unregistered [ 1242.246398] LNet: 64708:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1242.248925] LNetError: 64708:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1242.257424] LNet: Removed LNI 192.168.203.111@tcp [ 1242.597167] Key type .llcrypt unregistered [ 1242.599041] Key type ._llcrypt unregistered [ 1252.605504] Key type ._llcrypt registered [ 1252.607040] Key type .llcrypt registered [ 1252.658937] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 1253.033969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1253.075326] alg: No test for adler32 (adler32-zlib) [ 1253.942524] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 1254.038076] LNet: Added LNI 192.168.203.111@tcp [8/256/0/180] [ 1255.632342] Key type lgssc registered [ 1256.061362] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1258.861916] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1258.873247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1260.043536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1261.488893] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug +malloc all [ 1263.475710] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1266.108274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1266.235209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1268.519438] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug +malloc all [ 1270.309639] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:35 to 0x240000401:65) [ 1270.646188] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1280.482303] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1280.482436] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1280.490622] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1281.451329] LustreError: 67523:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 1281.487976] Lustre: server umount lustre-OST0000 complete [ 1282.932756] LustreError: 67726:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 1282.934686] LustreError: 67726:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1282.982643] Lustre: server umount lustre-MDT0000 complete [ 1286.864395] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing unload_modules_local [ 1287.987460] Key type lgssc unregistered [ 1288.138656] LNet: 68258:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1288.141722] LNetError: 68258:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1288.154290] LNet: Removed LNI 192.168.203.111@tcp [ 1288.469319] Key type .llcrypt unregistered [ 1288.471298] Key type ._llcrypt unregistered [ 1298.490380] Key type ._llcrypt registered [ 1298.491803] Key type .llcrypt registered [ 1298.531842] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 1298.979047] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1298.985678] alg: No test for adler32 (adler32-zlib) [ 1299.863318] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 1299.972585] LNet: Added LNI 192.168.203.111@tcp [8/256/0/180] [ 1301.584646] Key type lgssc registered [ 1302.043258] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1305.320247] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1305.329047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1320.800290] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1325.920264] LustreError: 69608:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.111@tcp: failed processing log, type 4: rc = -110 [ 1352.544272] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1354.762481] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 1356.942898] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1359.988859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1360.033657] Lustre: *** cfs_fail_loc=706, val=0*** [ 1360.174217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1360.198816] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:35 to 0x240000401:97) [ 1361.627776] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 1363.606132] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1365.473875] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1365.473879] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1365.481777] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1366.447130] LustreError: 71131:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 1366.484994] Lustre: server umount lustre-OST0000 complete [ 1367.971719] LustreError: 71331:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 1367.973476] LustreError: 71331:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1368.025488] Lustre: server umount lustre-MDT0000 complete [ 1371.970376] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing unload_modules_local [ 1373.189254] Key type lgssc unregistered [ 1373.333881] LNet: 71865:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1373.338068] LNetError: 71865:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1373.347623] LNet: Removed LNI 192.168.203.111@tcp [ 1373.675183] Key type .llcrypt unregistered [ 1373.676936] Key type ._llcrypt unregistered [ 1384.563031] Key type ._llcrypt registered [ 1384.564550] Key type .llcrypt registered [ 1384.603844] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 1385.044979] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1385.053542] alg: No test for adler32 (adler32-zlib) [ 1385.920542] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 1386.032886] LNet: Added LNI 192.168.203.111@tcp [8/256/0/180] [ 1387.616180] Key type lgssc registered [ 1388.034489] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1391.250342] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1391.258040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1393.809865] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 1396.773562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1396.927616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1399.063590] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 1401.984426] LustreError: MGC192.168.203.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1401.990292] Lustre: MGS: Client 6e7fb088-f5ce-44bf-9e2c-283a827e4003 (at 0@lo) reconnecting [ 1402.122541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1402.145669] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:35 to 0x240000401:129) [ 1403.496587] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 1412.577710] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1412.577719] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1412.587847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1417.697688] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1418.477102] LustreError: 75166:0:(obd_class.h:479:obd_check_dev()) Device 6 not setup [ 1418.512336] Lustre: server umount lustre-OST0000 complete [ 1419.912786] LustreError: 75370:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 1419.915644] LustreError: 75370:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1419.951905] Lustre: server umount lustre-MDT0000 complete [ 1421.419563] LustreError: 75572:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1421.424361] LustreError: 75572:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1421.453261] Lustre: server umount MGS complete [ 1424.573481] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 09:38:01 (1773236281) [ 1432.724746] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_hostid [ 1435.882956] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 1439.347871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1441.439334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1443.722092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1446.198027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1446.224439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1447.681307] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 1452.081016] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1452.112199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1452.208367] Lustre: lustre-OST0000: new disk, initializing [ 1452.210647] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1452.236664] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1454.354995] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 1457.568930] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1459.961442] LustreError: MGC192.168.203.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1459.966465] Lustre: MGS: Client ceec8f2b-2dcd-42a0-9089-d51055e9943c (at 0@lo) reconnecting [ 1460.028672] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1460.039152] LustreError: 78643:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 1460.055542] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1460.094421] Lustre: lustre-MDT0000: new disk, initializing [ 1460.138179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1460.148687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1460.190197] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1461.216816] Lustre: 80297:0:(client.c:1627:after_reply()) @@@ resending request on EINPROGRESS req@00000000376ff8dc x1859372966757760/t0(0) o700->lustre-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1773236334 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1461.562396] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 1463.269664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1463.274212] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1463.286638] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1480.672769] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1480.675341] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1480.679957] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1483.233466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1486.320114] LustreError: 80821:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 1486.322315] LustreError: 80821:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1486.357280] Lustre: server umount lustre-OST0000 complete [ 1502.176121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1502.209767] LustreError: 81023:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 1502.211621] LustreError: 81023:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1502.243786] Lustre: server umount lustre-MDT0000 complete [ 1503.594850] Lustre: server umount MGS complete [ 1506.784991] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 09:39:23 (1773236363) [ 1512.232538] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing unload_modules_local [ 1513.366828] Key type lgssc unregistered [ 1513.502466] LNet: 82256:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1513.505491] LNetError: 82256:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1513.515507] LNet: Removed LNI 192.168.203.111@tcp [ 1513.833210] Key type .llcrypt unregistered [ 1513.834696] Key type ._llcrypt unregistered [ 1521.440949] Key type ._llcrypt registered [ 1521.442271] Key type .llcrypt registered [ 1521.487389] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 1521.922586] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1521.943423] alg: No test for adler32 (adler32-zlib) [ 1522.797267] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 1522.880899] LNet: Added LNI 192.168.203.111@tcp [8/256/0/180] [ 1524.464146] Key type lgssc registered [ 1524.832693] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1527.543487] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1527.551385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1528.718430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1529.995903] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 1531.822178] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1534.166863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1534.282795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1536.290886] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 1538.202479] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1538.406099] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:33) [ 1543.649189] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1543.649819] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1543.660279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1547.952125] LustreError: 84878:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 1547.998484] Lustre: server umount lustre-OST0000 complete [ 1549.641947] LustreError: 85083:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 1549.644960] LustreError: 85083:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1549.724575] Lustre: server umount lustre-MDT0000 complete [ 1554.438779] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing unload_modules_local [ 1555.675907] Key type lgssc unregistered [ 1555.828611] LNet: 85614:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1555.832350] LNetError: 85614:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1555.842327] LNet: Removed LNI 192.168.203.111@tcp [ 1556.167482] Key type .llcrypt unregistered [ 1556.168832] Key type ._llcrypt unregistered [ 1563.443367] Key type ._llcrypt registered [ 1563.446161] Key type .llcrypt registered [ 1563.498760] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 1563.971416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1563.988295] alg: No test for adler32 (adler32-zlib) [ 1564.854028] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 1564.955320] LNet: Added LNI 192.168.203.111@tcp [8/256/0/180] [ 1566.544147] Key type lgssc registered [ 1566.942743] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1569.320223] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1569.321167] LustreError: 86653:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 1569.325368] Lustre: Skipped 1 previous similar message [ 1569.330174] LustreError: 86652:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 1569.333328] LustreError: 86653:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4995 [ 1569.336127] LustreError: 86653:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1569.340890] LustreError: 86653:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1569.342541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1570.513768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1571.979629] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing lsmod [ 1573.138176] LustreError: 87131:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 1573.140219] LustreError: 87132:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 1573.143542] LustreError: 87131:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4997 [ 1573.146540] LustreError: 87131:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1573.149788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1573.151292] LustreError: 87131:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1573.258080] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1574.433815] LustreError: 87317:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 1574.485384] Lustre: server umount lustre-MDT0000 complete [ 1575.999936] LustreError: 86672:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773236433 with bad export cookie 5071775011058276826 [ 1576.004323] LustreError: MGC192.168.203.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1586.170118] LustreError: 87517:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1586.172700] LustreError: 87517:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1586.209487] Lustre: server umount lustre-OST0000 complete [ 1589.362634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1589.538062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1590.930260] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 1592.930360] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1595.657528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1595.806345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1598.171316] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 1599.909890] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:65) [ 1600.456373] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1605.090207] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1605.090419] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1605.096415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1609.263211] LustreError: 89440:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 1609.265830] LustreError: 89440:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1609.300356] Lustre: server umount lustre-OST0000 complete [ 1610.813186] Lustre: server umount lustre-MDT0000 complete [ 1614.796936] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing unload_modules_local [ 1615.906415] Key type lgssc unregistered [ 1616.041198] LNet: 90173:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1616.046170] LNetError: 90173:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1616.055581] LNet: Removed LNI 192.168.203.111@tcp [ 1616.361161] Key type .llcrypt unregistered [ 1616.363182] Key type ._llcrypt unregistered [ 1625.997840] Key type ._llcrypt registered [ 1625.999024] Key type .llcrypt registered [ 1626.048266] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 1626.373491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1626.383573] alg: No test for adler32 (adler32-zlib) [ 1627.298556] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 1627.404710] LNet: Added LNI 192.168.203.111@tcp [8/256/0/180] [ 1628.992164] Key type lgssc registered [ 1629.420059] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1632.253518] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1632.267802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1633.450269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1634.906246] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 1636.865671] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1639.414974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1639.538460] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1641.624037] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 1643.624457] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:67 to 0x240000400:97) [ 1643.949383] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1647.035951] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1647.962272] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1654.242128] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1654.242488] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1654.253814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1655.277139] LustreError: 93093:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 1655.329677] Lustre: server umount lustre-OST0000 complete [ 1656.869841] LustreError: 93296:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 1656.872209] LustreError: 93296:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1656.931328] Lustre: server umount lustre-MDT0000 complete [ 1660.927535] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing unload_modules_local [ 1662.126343] Key type lgssc unregistered [ 1662.276625] LNet: 93828:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1662.280918] LNetError: 93828:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1662.290383] LNet: Removed LNI 192.168.203.111@tcp [ 1662.650661] Key type .llcrypt unregistered [ 1662.652493] Key type ._llcrypt unregistered [ 1670.027836] Key type ._llcrypt registered [ 1670.028778] Key type .llcrypt registered [ 1670.073396] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 1670.431375] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1670.468205] alg: No test for adler32 (adler32-zlib) [ 1671.343469] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 1671.436749] LNet: Added LNI 192.168.203.111@tcp [8/256/0/180] [ 1673.032185] Key type lgssc registered [ 1673.533641] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1676.631846] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1676.639931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1677.845312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1679.216718] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 1681.130073] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1683.713089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1683.837582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1685.846691] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 1687.818467] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1687.976683] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 1693.153889] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1693.154340] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1693.163110] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1697.581180] LustreError: 96453:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 1697.617278] Lustre: server umount lustre-OST0000 complete [ 1698.954366] LustreError: 96655:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 1698.957968] LustreError: 96655:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1699.011343] Lustre: server umount lustre-MDT0000 complete [ 1702.617520] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing unload_modules_local [ 1703.662115] Key type lgssc unregistered [ 1703.784364] LNet: 97187:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1703.786790] LNetError: 97187:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1703.797332] LNet: Removed LNI 192.168.203.111@tcp [ 1704.084602] Key type .llcrypt unregistered [ 1704.085640] Key type ._llcrypt unregistered [ 1713.189166] Key type ._llcrypt registered [ 1713.190646] Key type .llcrypt registered [ 1713.230521] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 1713.641458] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1713.663544] alg: No test for adler32 (adler32-zlib) [ 1714.519172] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 1714.619350] LNet: Added LNI 192.168.203.111@tcp [8/256/0/180] [ 1716.208148] Key type lgssc registered [ 1716.630356] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1719.391774] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1719.398588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1720.552162] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1721.843525] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 1723.765636] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1726.237254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1726.381852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1728.543467] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 1730.469831] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:131 to 0x240000400:161) [ 1730.589398] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1734.296390] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 1735.731333] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 1735.733990] Lustre: Skipped 1 previous similar message [ 1735.874346] Lustre: lustre-MDT0000: root_squash is set to 0:0 [ 1735.883177] Lustre: lustre-MDT0000: nosquash_nids is cleared [ 1737.295608] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 1737.298732] Lustre: Skipped 1 previous similar message [ 1744.887885] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 1747.581352] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.11@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 1747.585033] Lustre: Skipped 1 previous similar message [ 1751.480402] Lustre: lustre-MDT0000: nosquash_nids set to 2@gni 0@lo 192.168.203.11@tcp 192.168.0.[2,10]@tcp [ 1758.952949] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.11/24@tcp in log lustre-MDT0000 [ 1758.956794] Lustre: Skipped 1 previous similar message [ 1759.157483] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.11/24@tcp [ 1766.369575] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1766.369642] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1766.376727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1771.489407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1771.504077] LustreError: 101443:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 1771.532559] Lustre: server umount lustre-OST0000 complete [ 1772.768555] LustreError: 101644:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 1772.772045] LustreError: 101644:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1772.817607] Lustre: server umount lustre-MDT0000 complete [ 1776.271583] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing unload_modules_local [ 1777.262237] Key type lgssc unregistered [ 1777.394390] LNet: 102174:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1777.398543] LNetError: 102174:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1777.411369] LNet: Removed LNI 192.168.203.111@tcp [ 1777.689400] Key type .llcrypt unregistered [ 1777.690915] Key type ._llcrypt unregistered [ 1788.088846] Key type ._llcrypt registered [ 1788.090288] Key type .llcrypt registered [ 1788.131549] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 1788.501637] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1788.510808] alg: No test for adler32 (adler32-zlib) [ 1789.383263] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 1789.477205] LNet: Added LNI 192.168.203.111@tcp [8/256/0/180] [ 1791.064187] Key type lgssc registered [ 1791.421296] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1794.134559] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1794.141337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1795.286165] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 1795.289625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1795.295875] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.11/24@tcp [ 1796.593257] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 1798.429202] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1800.766067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1800.898453] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1802.809895] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 1804.651936] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1805.030784] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:169 to 0x240000400:193) [ 1810.401256] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1810.402341] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1810.409295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1811.825116] LustreError: 105188:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 1811.857256] Lustre: server umount lustre-OST0000 complete [ 1815.230764] LustreError: 105388:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 1815.233801] LustreError: 105388:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1815.290668] Lustre: server umount lustre-MDT0000 complete [ 1818.923675] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing unload_modules_local [ 1819.978946] Key type lgssc unregistered [ 1820.122645] LNet: 105921:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1820.125028] LNetError: 105921:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1820.133269] LNet: Removed LNI 192.168.203.111@tcp [ 1820.448470] Key type .llcrypt unregistered [ 1820.449978] Key type ._llcrypt unregistered [ 1828.839829] Key type ._llcrypt registered [ 1828.840802] Key type .llcrypt registered [ 1828.878536] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_hostid [ 1834.847343] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 1835.158872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1835.166373] alg: No test for adler32 (adler32-zlib) [ 1836.027531] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 1836.105435] LNet: Added LNI 192.168.203.111@tcp [8/256/0/180] [ 1837.688130] Key type lgssc registered [ 1838.122852] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1841.233897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1843.205591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1845.268359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1847.563212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1847.585352] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1847.593056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1848.688055] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1848.701503] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1848.738693] Lustre: lustre-MDT0000: new disk, initializing [ 1848.767989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1848.774865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1850.137114] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 1853.155425] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1855.513315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1855.539184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1855.641719] Lustre: lustre-OST0000: new disk, initializing [ 1855.644566] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1855.646379] Lustre: Skipped 1 previous similar message [ 1855.671165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1857.474029] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1857.477250] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1857.486563] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1857.511947] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 1860.601104] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1861.839212] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1861.910644] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1862.625996] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1862.626504] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1862.632219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1867.745475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1868.590120] LustreError: 110672:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 1868.623286] Lustre: server umount lustre-OST0000 complete [ 1869.892293] LustreError: 110874:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 1869.894541] LustreError: 110874:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1869.951467] Lustre: server umount lustre-MDT0000 complete [ 1872.750458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1872.908435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1874.140200] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 1875.870532] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1878.055683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1878.164879] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1880.020903] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 1881.788760] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1883.050812] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1884.136313] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 1887.431321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1887.455842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1887.503976] Lustre: lustre-OST0001: new disk, initializing [ 1887.505628] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1887.527900] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1889.295831] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 1892.372321] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 1894.911706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 1894.914684] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 1894.923421] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 1895.508137] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 1896.676332] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9696d0aaf800.ost_server_uuid 50 [ 1897.212253] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9696d0aaf800.ost_server_uuid in FULL state after 0 sec [ 1900.002628] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1900.006260] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1900.012798] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 1903.010708] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 1904.941094] LustreError: 114322:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1904.943102] LustreError: 114322:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1904.957946] Lustre: server umount lustre-OST0001 complete [ 1908.194085] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1908.197592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1912.299143] LustreError: 114525:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 1912.301092] LustreError: 114525:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1912.329519] Lustre: server umount lustre-OST0000 complete [ 1913.614099] Lustre: server umount lustre-MDT0000 complete [ 1918.954792] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing unload_modules_local [ 1919.950321] Key type lgssc unregistered [ 1920.087389] LNet: 115562:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1920.089693] LNetError: 115562:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1920.098304] LNet: Removed LNI 192.168.203.111@tcp [ 1920.392670] Key type .llcrypt unregistered [ 1920.393614] Key type ._llcrypt unregistered [ 1929.658037] Key type ._llcrypt registered [ 1929.658878] Key type .llcrypt registered [ 1929.694976] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_hostid [ 1935.847971] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 1936.164597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1936.177222] alg: No test for adler32 (adler32-zlib) [ 1937.041336] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 1937.138798] LNet: Added LNI 192.168.203.111@tcp [8/256/0/180] [ 1938.720149] Key type lgssc registered [ 1939.189872] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1942.670881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1944.917941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1947.113353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1949.566295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1949.580083] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1949.590652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1950.678594] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1950.690343] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1950.727496] Lustre: lustre-MDT0000: new disk, initializing [ 1950.754121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1950.761842] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1952.046771] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 1955.258607] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1958.009472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1958.040232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1958.157503] Lustre: lustre-OST0000: new disk, initializing [ 1958.160083] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1958.162618] Lustre: Skipped 1 previous similar message [ 1958.190165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1960.366479] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 1963.609911] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1964.039246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1964.044180] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1964.057339] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1965.086512] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1965.162786] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1969.122036] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1969.122077] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1969.130642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1971.888351] LustreError: 120685:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 1971.924913] Lustre: server umount lustre-OST0000 complete [ 1973.377941] LustreError: 120887:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 1973.380435] LustreError: 120887:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1973.447465] Lustre: server umount lustre-MDT0000 complete [ 1976.489406] Lustre: DEBUG MARKER: == conf-sanity test 47a: server restart does not lose lru_resize ========================================================== 09:47:13 (1773236833) [ 1981.013697] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_hostid [ 1984.017911] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 1987.058527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1988.999994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1991.051290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1993.363577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1993.385947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1993.465983] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1993.478430] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1993.517242] Lustre: lustre-MDT0000: new disk, initializing [ 1993.543339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1993.549784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1994.852787] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 1998.043606] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2000.365203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2000.385530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2000.469500] Lustre: lustre-OST0000: new disk, initializing [ 2000.471176] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2000.472890] Lustre: Skipped 1 previous similar message [ 2000.493688] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2002.386751] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 2005.465527] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2007.042260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2007.046158] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2007.056102] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 2012.735464] Lustre: Failing over lustre-OST0000 [ 2012.768125] LustreError: 125032:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 2012.770302] LustreError: 125032:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2012.809301] Lustre: server umount lustre-OST0000 complete [ 2025.726894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2025.830996] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2025.834817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2025.842842] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2027.250914] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2027.752209] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 2029.077893] Lustre: Failing over lustre-MDT0000 [ 2029.142878] LustreError: 126002:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 2029.145339] LustreError: 126002:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2029.195959] Lustre: server umount lustre-MDT0000 complete [ 2042.365447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2042.427970] LustreError: MGC192.168.203.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2042.541763] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2042.541938] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2042.562114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2042.579921] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2042.608746] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2042.617792] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2043.916880] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 2046.432129] Lustre: 117343:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773236888/real 1773236888] req@000000008066fcf6 x1859373544535168/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773236904 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2047.970981] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2047.975387] Lustre: lustre-OST0000: Recovery over after 0:20, of 2 clients 2 recovered and 0 were evicted. [ 2047.977592] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:33) [ 2053.089543] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2053.089598] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2053.093546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2054.959094] LustreError: 126900:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 2054.961820] LustreError: 126900:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2054.995111] Lustre: server umount lustre-OST0000 complete [ 2056.413214] Lustre: server umount lustre-MDT0000 complete [ 2060.291941] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing unload_modules_local [ 2061.382098] Key type lgssc unregistered [ 2061.519343] LNet: 127633:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2061.522306] LNetError: 127633:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2061.532303] LNet: Removed LNI 192.168.203.111@tcp [ 2061.836490] Key type .llcrypt unregistered [ 2061.838133] Key type ._llcrypt unregistered [ 2081.799533] Key type ._llcrypt registered [ 2081.800420] Key type .llcrypt registered [ 2081.834048] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 2082.163543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2082.196043] alg: No test for adler32 (adler32-zlib) [ 2083.043554] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 2083.136958] LNet: Added LNI 192.168.203.111@tcp [8/256/0/180] [ 2084.720134] Key type lgssc registered [ 2085.124628] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2087.912115] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2087.919868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2089.094020] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2090.495246] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 2093.721366] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2095.834291] LustreError: 129675:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 2095.890798] Lustre: server umount lustre-MDT0000 complete [ 2100.342977] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_hostid [ 2103.745796] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 2107.144247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2109.251178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2111.397927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2113.873751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2113.900163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2113.991734] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2114.006223] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2114.042942] Lustre: lustre-MDT0000: new disk, initializing [ 2114.071736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2114.079056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2115.468871] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 2118.586599] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2120.863978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2120.893162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2120.984340] Lustre: lustre-OST0000: new disk, initializing [ 2120.987185] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2120.989847] Lustre: Skipped 1 previous similar message [ 2121.013844] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2122.886835] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 2123.901843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2123.904910] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2123.912438] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 2125.999373] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2127.852237] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2128.433274] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 09:49:45 (1773236985) [ 2211.296702] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2211.301082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2215.674884] LustreError: 134535:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 2215.676853] LustreError: 134535:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2215.730563] Lustre: server umount lustre-MDT0000 complete [ 2217.124673] LustreError: 131888:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773237074 with bad export cookie 16955499254369276483 [ 2217.126501] LustreError: MGC192.168.203.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2217.129352] LustreError: 131888:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2223.210979] LustreError: 134740:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2223.213797] LustreError: 134740:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2223.249102] Lustre: server umount lustre-OST0000 complete [ 2226.163799] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_hostid [ 2229.182568] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 2232.530848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2234.561776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2236.719402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2239.058204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2239.084706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2239.171502] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2239.182668] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2239.220322] Lustre: lustre-MDT0000: new disk, initializing [ 2239.245591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2239.253274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2240.580421] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 2243.770872] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2246.121641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2246.142184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2246.233597] Lustre: lustre-OST0000: new disk, initializing [ 2246.235553] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2246.238111] Lustre: Skipped 1 previous similar message [ 2246.262227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2248.203683] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 2248.772212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2248.775437] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2248.787508] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 2251.484817] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2252.886313] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2252.963993] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2253.793339] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2253.793412] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2253.803100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2258.913637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2259.759145] LustreError: 138634:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 2259.761656] LustreError: 138634:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2259.797173] Lustre: server umount lustre-OST0000 complete [ 2261.169094] Lustre: server umount lustre-MDT0000 complete [ 2264.129268] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 09:52:01 (1773237121) [ 2268.606940] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_hostid [ 2271.399489] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 2274.464091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2276.455278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2278.520899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2280.770765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2280.793677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2280.878410] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2280.890928] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2280.930155] Lustre: lustre-MDT0000: new disk, initializing [ 2280.961158] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2280.971391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2282.367193] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 2285.595758] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2288.019085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2288.044763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2289.279062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2289.284436] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2289.293684] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 2290.133552] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 2293.328316] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2299.873557] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2299.873926] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2299.886245] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2301.356098] LustreError: 143081:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 2301.357912] LustreError: 143081:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2301.392949] Lustre: server umount lustre-OST0000 complete [ 2302.854636] Lustre: server umount lustre-MDT0000 complete [ 2305.730307] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 09:52:42 (1773237162) [ 2310.098933] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_hostid [ 2313.014779] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 2316.089793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2318.017429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2320.190914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2322.506674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2322.533255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2322.612064] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2322.623530] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2322.625284] Lustre: Skipped 2 previous similar messages [ 2322.659915] Lustre: lustre-MDT0000: new disk, initializing [ 2322.661319] Lustre: Skipped 1 previous similar message [ 2322.684637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2322.686706] Lustre: Skipped 1 previous similar message [ 2322.692335] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2323.949316] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 2326.999442] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2329.266537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2329.288732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2329.384192] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 2331.288546] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 2334.329956] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2334.704917] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2334.715637] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 2344.929964] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2344.930056] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2344.938271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2347.310020] LustreError: 147535:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 2347.313292] LustreError: 147535:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2347.354872] Lustre: server umount lustre-OST0000 complete [ 2348.736799] Lustre: server umount lustre-MDT0000 complete [ 2352.321150] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing unload_modules_local [ 2353.344987] Key type lgssc unregistered [ 2353.468388] LNet: 148267:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2353.471598] LNetError: 148267:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2353.481327] LNet: Removed LNI 192.168.203.111@tcp [ 2353.775316] Key type .llcrypt unregistered [ 2353.776361] Key type ._llcrypt unregistered [ 2362.559195] Key type ._llcrypt registered [ 2362.560151] Key type .llcrypt registered [ 2362.604757] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 2362.955470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2362.997717] alg: No test for adler32 (adler32-zlib) [ 2363.847881] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 2363.926733] LNet: Added LNI 192.168.203.111@tcp [8/256/0/180] [ 2365.504139] Key type lgssc registered [ 2365.869429] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2368.385903] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2368.392461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2369.555762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2370.863215] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 2372.677789] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2374.991860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2375.117058] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2377.080110] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 2378.879439] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2380.261147] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:33) [ 2400.737402] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2400.737862] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2400.741434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2402.604209] LustreError: 151108:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 2402.632801] Lustre: server umount lustre-OST0000 complete [ 2403.938112] LustreError: 151310:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 2403.940351] LustreError: 151310:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2404.000547] Lustre: server umount lustre-MDT0000 complete [ 2407.397407] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing unload_modules_local [ 2408.362424] Key type lgssc unregistered [ 2408.489414] LNet: 151840:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2408.491848] LNetError: 151840:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2408.500260] LNet: Removed LNI 192.168.203.111@tcp [ 2408.772986] Key type .llcrypt unregistered [ 2408.774027] Key type ._llcrypt unregistered [ 2416.975713] Key type ._llcrypt registered [ 2416.976663] Key type .llcrypt registered [ 2417.011125] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 2417.354628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2417.405638] alg: No test for adler32 (adler32-zlib) [ 2418.255138] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 2418.339714] LNet: Added LNI 192.168.203.111@tcp [8/256/0/180] [ 2419.920170] Key type lgssc registered [ 2420.247422] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2422.682742] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2422.689290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2423.849298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2425.048138] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 2426.583497] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2428.619299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2428.756176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2430.448316] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 2432.065113] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2433.892857] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:35 to 0x240000400:65) [ 2443.503209] Lustre: lustre-OST0000: Not available for connect from 192.168.203.11@tcp (stopping) [ 2444.257702] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2444.257797] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2444.265183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2446.449123] LustreError: 154657:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 2446.482386] Lustre: server umount lustre-OST0000 complete [ 2448.264322] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9696c62c3800.ost_server_uuid 50 [ 2448.773260] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9696c62c3800.ost_server_uuid in DISCONN state after 0 sec [ 2449.276783] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 2465.337673] LustreError: 155053:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 2465.339421] LustreError: 155053:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2465.383582] Lustre: server umount lustre-MDT0000 complete [ 2467.774204] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 09:55:25 (1773237325) [ 2469.792084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2469.937564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2471.119935] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 2472.845603] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2474.999032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2475.112320] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2476.919327] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 2478.756337] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2480.229057] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:67 to 0x240000400:97) [ 2480.911968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2480.936460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2480.990525] Lustre: lustre-OST0001: new disk, initializing [ 2480.992537] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2481.017071] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2482.776360] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 2485.708218] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2486.282235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 2486.285691] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 2486.293789] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 2490.848921] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2490.852334] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2490.856227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2492.591106] LustreError: 158119:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 2492.593036] LustreError: 158119:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2492.607498] Lustre: server umount lustre-OST0000 complete [ 2494.661816] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2494.734731] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2501.353260] LustreError: 156527:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2501.358447] LustreError: 156527:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2506.721282] LustreError: 156527:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2506.721581] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2506.726090] LustreError: 156527:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2506.728547] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2506.736785] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2506.738664] Lustre: Skipped 2 previous similar messages [ 2510.700533] LustreError: 158573:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2510.703512] LustreError: 158573:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2510.737444] Lustre: server umount lustre-OST0001 complete [ 2512.061756] Lustre: server umount lustre-MDT0000 complete [ 2513.883740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2515.113759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2516.396511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2518.436225] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 09:56:15 (1773237375) [ 2520.438920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2520.524194] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2520.535650] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2520.599891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2521.776463] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 2523.414080] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2525.506639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2525.568773] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2525.603773] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 2527.257641] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 2528.814385] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2530.788636] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 2530.916591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2530.938842] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2532.748370] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 2534.423970] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2535.904795] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2535.907075] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2535.910678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2537.259071] LustreError: 162431:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 2537.261580] LustreError: 162431:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 2537.275600] Lustre: server umount lustre-OST0000 complete [ 2539.757930] LustreError: 161758:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2542.049669] LustreError: 161758:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2547.170515] LustreError: 161758:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2547.176653] LustreError: 161758:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2552.290150] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2552.292950] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2552.296582] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2552.299349] Lustre: Skipped 1 previous similar message [ 2553.902096] LustreError: 162635:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2553.904575] LustreError: 162635:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2553.937104] Lustre: server umount lustre-OST0001 complete [ 2555.236592] Lustre: server umount lustre-MDT0000 complete [ 2557.027023] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2558.217886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2559.375617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2561.428984] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 09:56:58 (1773237418) [ 2565.299116] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_hostid [ 2567.934586] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 2570.926297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2572.769686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2574.704393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2576.820045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2576.844577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2576.911915] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2576.920239] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2576.952907] Lustre: lustre-MDT0000: new disk, initializing [ 2576.971457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2576.973144] Lustre: Skipped 2 previous similar messages [ 2576.978169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2578.190207] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 2581.349065] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2583.599055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2583.619123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2583.716615] Lustre: lustre-OST0000: new disk, initializing [ 2583.718622] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2583.720346] Lustre: Skipped 1 previous similar message [ 2585.667635] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 2585.795601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2585.798817] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2585.808609] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 2588.828541] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2590.192345] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2590.268291] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2591.201771] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2591.202069] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2591.210281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2596.971517] LustreError: 167849:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 2596.973382] LustreError: 167849:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 2597.007743] Lustre: server umount lustre-OST0000 complete [ 2598.467835] Lustre: server umount lustre-MDT0000 complete [ 2601.351059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2602.945824] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 2604.783734] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2607.078590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2609.137364] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 2610.963742] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2612.387392] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2612.469557] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2617.826187] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2617.826244] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2619.211550] Lustre: server umount lustre-OST0000 complete [ 2621.203382] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2621.280319] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2644.713177] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2644.822903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2644.825608] Lustre: Skipped 3 previous similar messages [ 2646.511299] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 2648.110948] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2648.644600] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 2655.595180] LustreError: 171637:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 2655.596737] LustreError: 171637:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 2655.628267] Lustre: server umount lustre-OST0000 complete [ 2656.891856] Lustre: server umount lustre-MDT0000 complete [ 2659.469640] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 09:58:36 (1773237516) [ 2661.467102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2662.718227] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 2664.455701] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2666.713026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2668.727286] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 2670.502768] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2671.989259] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2672.083716] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2674.367735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2674.396696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2674.446273] Lustre: lustre-OST0001: new disk, initializing [ 2674.448131] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2676.401682] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 2679.606499] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2680.942722] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2683.398553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 2683.401362] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 2683.410382] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 2684.078817] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 2688.482791] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2688.486101] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2688.489217] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2688.490490] Lustre: Skipped 2 previous similar messages [ 2690.813872] Lustre: server umount lustre-OST0001 complete [ 2692.729683] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2692.806461] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2692.918635] LustreError: 173315:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2692.922531] LustreError: 173315:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2708.961863] LustreError: 173314:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2708.968269] LustreError: 173314:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 2716.589084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2716.664079] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2716.667207] Lustre: Skipped 3 previous similar messages [ 2718.722234] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 2720.676320] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2721.254706] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 2721.763573] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 2721.769623] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2724.014228] LustreError: 176827:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2724.017042] LustreError: 176827:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 2724.030219] Lustre: server umount lustre-OST0001 complete [ 2741.217748] LustreError: 174082:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2741.222299] LustreError: 174082:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2742.296145] Lustre: server umount lustre-OST0000 complete [ 2749.940489] Lustre: server umount lustre-MDT0000 complete [ 2751.909790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2753.247352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2754.643304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2756.851737] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 10:00:14 (1773237614) [ 2759.034976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2759.125696] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2759.136160] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2760.530871] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 2762.393701] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2764.770208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2764.837163] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2766.884199] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 2768.733447] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2772.613868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2772.635638] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2774.738253] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 2776.708306] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2779.317428] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 2780.486098] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2780.555717] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2781.617693] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9696c763d000.ost_server_uuid 50 [ 2782.076926] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9696c763d000.ost_server_uuid in FULL state after 0 sec [ 2782.444805] Lustre: Permanently deactivating lustre-OST0001 [ 2782.448672] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 2782.524889] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 2783.206243] Lustre: Permanently reactivating lustre-OST0001 [ 2787.823901] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2787.828497] Lustre: Skipped 2 previous similar messages [ 2790.140161] Lustre: server umount lustre-OST0001 complete [ 2797.445036] Lustre: server umount lustre-OST0000 complete [ 2808.288179] LustreError: 181456:0:(osp_dev.c:692:osp_process_config()) lustre-OST0001-osc-MDT0000: unknown param osc.active=1 [ 2808.423613] Lustre: server umount lustre-MDT0000 complete [ 2810.415646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2811.728911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2812.906698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2814.824459] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 10:01:12 (1773237672) [ 2815.169149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2817.171756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2817.251403] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2817.261782] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2817.268497] Lustre: Skipped 3 previous similar messages [ 2818.471123] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 2820.114335] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2822.347846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2822.373539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2822.457241] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2822.463592] Lustre: Permanently deactivating lustre-OST0000 [ 2822.499868] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 2822.499962] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2822.501677] Lustre: Skipped 5 previous similar messages [ 2824.333525] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 2825.948832] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2828.075346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2829.835268] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 2830.971239] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:35 to 0x280000400:65) [ 2831.441106] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2831.976108] Lustre: Permanently reactivating lustre-OST0000 [ 2831.980512] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 2831.982193] Lustre: Skipped 2 previous similar messages [ 2836.985575] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 2846.692951] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2846.695027] LustreError: Skipped 2 previous similar messages [ 2850.240670] Lustre: server umount lustre-OST0001 complete [ 2851.809313] LustreError: 184030:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2851.813273] LustreError: 184030:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2857.581202] LustreError: 185861:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 2857.583290] LustreError: 185861:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 2857.615205] Lustre: server umount lustre-OST0000 complete [ 2858.976043] Lustre: server umount lustre-MDT0000 complete [ 2862.514916] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing unload_modules_local [ 2863.545153] Key type lgssc unregistered [ 2863.682369] LNet: 186595:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2863.684604] LNetError: 186595:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2863.692307] LNet: Removed LNI 192.168.203.111@tcp [ 2863.970883] Key type .llcrypt unregistered [ 2863.971800] Key type ._llcrypt unregistered [ 2871.402633] Key type ._llcrypt registered [ 2871.404026] Key type .llcrypt registered [ 2871.437976] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_hostid [ 2876.821705] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 2877.092856] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2877.099061] alg: No test for adler32 (adler32-zlib) [ 2877.939593] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 2878.013207] LNet: Added LNI 192.168.203.111@tcp [8/256/0/180] [ 2879.584133] Key type lgssc registered [ 2879.889271] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2882.384142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2884.346689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2886.430270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2888.536083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2888.551825] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2888.555815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2889.629793] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2889.640290] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2889.671703] Lustre: lustre-MDT0000: new disk, initializing [ 2889.691753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2889.698613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2891.095407] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 2894.183901] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2896.514256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2896.546303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2896.666537] Lustre: lustre-OST0000: new disk, initializing [ 2896.668666] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2896.670564] Lustre: Skipped 1 previous similar message [ 2896.697946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2897.275602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2897.278406] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2897.286495] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 2898.539670] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 2901.455628] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2903.137788] LustreError: 190468:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 2907.270266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2907.293347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2907.343584] Lustre: lustre-OST0001: new disk, initializing [ 2907.344958] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2907.365208] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2909.137202] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 2912.067695] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2912.577734] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 2913.232071] LustreError: 190468:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 2913.238854] LustreError: 190468:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 2915.333299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 2915.337280] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 2915.346400] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 2923.328146] LustreError: 190468:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 2925.541394] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2925.541450] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2925.545042] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2930.657706] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2935.778261] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2938.336094] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2938.351844] LustreError: 192215:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2938.367506] Lustre: server umount lustre-OST0001 complete [ 2946.017881] LustreError: 190442:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2946.025835] LustreError: 190442:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2951.137959] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2951.138016] LustreError: 190442:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2951.143080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2956.257497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2956.257975] LustreError: 190443:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2956.396122] LustreError: 192439:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 2956.398989] LustreError: 192439:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2956.437823] Lustre: server umount lustre-OST0000 complete [ 2963.907466] LustreError: 192641:0:(obd_class.h:479:obd_check_dev()) Device 17 not setup [ 2963.909566] LustreError: 192641:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2963.962946] Lustre: server umount lustre-MDT0000 complete [ 2967.673683] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing unload_modules_local [ 2968.709160] Key type lgssc unregistered [ 2968.838700] LNet: 193173:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2968.841312] LNetError: 193173:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2968.848341] LNet: Removed LNI 192.168.203.111@tcp [ 2969.146667] Key type .llcrypt unregistered [ 2969.148316] Key type ._llcrypt unregistered [ 2977.874881] Key type ._llcrypt registered [ 2977.875842] Key type .llcrypt registered [ 2977.913058] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_hostid [ 2984.131685] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 2984.538467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2984.560118] alg: No test for adler32 (adler32-zlib) [ 2985.417859] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 2985.508181] LNet: Added LNI 192.168.203.111@tcp [8/256/0/180] [ 2987.088160] Key type lgssc registered [ 2987.468267] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2990.470089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2992.489505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2994.725942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2996.996584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2997.018335] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2997.022704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2998.105459] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2998.116305] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2998.153859] Lustre: lustre-MDT0000: new disk, initializing [ 2998.177643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2998.184180] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2999.453990] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 3002.591382] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3004.841783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3004.866197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3004.955186] Lustre: lustre-OST0000: new disk, initializing [ 3004.956982] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3004.958927] Lustre: Skipped 1 previous similar message [ 3004.981486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3006.593160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3006.595772] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3006.605223] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3006.812821] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 3009.959305] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3011.372925] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3011.459901] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3017.186094] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3017.186179] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3017.194199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3022.305254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3026.400115] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3026.425692] LustreError: 198298:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 3026.463923] Lustre: server umount lustre-OST0000 complete [ 3027.749558] LustreError: 198501:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 3027.752374] LustreError: 198501:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3027.809704] Lustre: server umount lustre-MDT0000 complete [ 3030.395117] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 10:04:47 (1773237887) [ 3032.439637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3032.599458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3033.754968] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 3035.352516] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3037.478993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3037.597741] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3039.378777] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 3041.075751] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3088.865672] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3088.866039] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3088.872171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3090.861119] LustreError: 200696:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 3090.863356] LustreError: 200696:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3090.908143] Lustre: server umount lustre-OST0000 complete [ 3092.489355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3096.531791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3096.655299] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3098.451831] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 3100.186515] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3108.897841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3108.902237] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3134.433966] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3134.434036] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3134.438957] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3139.373894] LustreError: 202351:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 3139.376878] LustreError: 202351:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3139.420563] Lustre: server umount lustre-OST0000 complete [ 3140.756030] Lustre: server umount lustre-MDT0000 complete [ 3144.202832] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing unload_modules_local [ 3145.212786] Key type lgssc unregistered [ 3145.340426] LNet: 203085:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3145.343544] LNetError: 203085:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3145.353463] LNet: Removed LNI 192.168.203.111@tcp [ 3145.656419] Key type .llcrypt unregistered [ 3145.658243] Key type ._llcrypt unregistered [ 3154.423178] Key type ._llcrypt registered [ 3154.424629] Key type .llcrypt registered [ 3154.455566] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 3154.753472] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3154.801565] alg: No test for adler32 (adler32-zlib) [ 3155.651151] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 3155.732191] LNet: Added LNI 192.168.203.111@tcp [8/256/0/180] [ 3157.304113] Key type lgssc registered [ 3157.608631] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3159.974097] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3159.979883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3161.106133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3162.238882] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 3163.846684] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3165.898632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3165.998637] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3167.742912] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 3169.410986] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3171.108841] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:42 to 0x240000400:65) [ 3186.657675] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3186.657679] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3186.658429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3188.525097] LustreError: 206676:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 3188.558209] Lustre: server umount lustre-OST0000 complete [ 3189.761226] LustreError: 206879:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 3189.762857] LustreError: 206879:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3189.805788] Lustre: server umount lustre-MDT0000 complete [ 3193.069102] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing unload_modules_local [ 3194.028516] Key type lgssc unregistered [ 3194.135373] LNet: 207411:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3194.139658] LNetError: 207411:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3194.150285] LNet: Removed LNI 192.168.203.111@tcp [ 3194.411314] Key type .llcrypt unregistered [ 3194.412460] Key type ._llcrypt unregistered [ 3200.858076] Key type ._llcrypt registered [ 3200.859066] Key type .llcrypt registered [ 3200.894214] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 3201.275824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3201.280710] alg: No test for adler32 (adler32-zlib) [ 3202.117023] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 3202.189254] LNet: Added LNI 192.168.203.111@tcp [8/256/0/180] [ 3203.760124] Key type lgssc registered [ 3204.087674] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3206.457226] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3206.462671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3207.593974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3208.724782] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 3210.347888] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3212.403127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3212.494403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3214.192402] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 3215.860357] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3217.637473] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:67 to 0x240000400:97) [ 3228.129500] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3228.130208] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3228.137471] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3231.407112] LustreError: 210177:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 3231.438588] Lustre: server umount lustre-OST0000 complete [ 3232.779035] LustreError: 210380:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 3232.780649] LustreError: 210380:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3232.833234] Lustre: server umount lustre-MDT0000 complete [ 3236.472500] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing unload_modules_local [ 3237.508257] Key type lgssc unregistered [ 3237.635460] LNet: 210911:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3237.638925] LNetError: 210911:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3237.648491] LNet: Removed LNI 192.168.203.111@tcp [ 3237.956825] Key type .llcrypt unregistered [ 3237.957744] Key type ._llcrypt unregistered [ 3246.676486] Key type ._llcrypt registered [ 3246.677450] Key type .llcrypt registered [ 3246.718500] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 3247.075633] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3247.122183] alg: No test for adler32 (adler32-zlib) [ 3247.986255] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 3248.072100] LNet: Added LNI 192.168.203.111@tcp [8/256/0/180] [ 3249.656176] Key type lgssc registered [ 3250.031368] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3252.654870] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3252.662775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3253.823795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3255.099434] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 3256.877702] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3259.184353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3259.288414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3261.113497] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 3262.886409] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3263.398850] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 3278.818061] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3278.818182] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3278.824724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3282.478097] LustreError: 214501:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 3282.511991] Lustre: server umount lustre-OST0000 complete [ 3283.809883] LustreError: 214703:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 3283.811960] LustreError: 214703:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3283.865065] Lustre: server umount lustre-MDT0000 complete [ 3287.474252] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing unload_modules_local [ 3288.506636] Key type lgssc unregistered [ 3288.647437] LNet: 215234:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3288.650775] LNetError: 215234:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3288.658435] LNet: Removed LNI 192.168.203.111@tcp [ 3288.946080] Key type .llcrypt unregistered [ 3288.947086] Key type ._llcrypt unregistered [ 3295.566270] Key type ._llcrypt registered [ 3295.567393] Key type .llcrypt registered [ 3295.603716] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 3295.905895] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3295.919189] alg: No test for adler32 (adler32-zlib) [ 3296.779021] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 3296.871350] LNet: Added LNI 192.168.203.111@tcp [8/256/0/180] [ 3298.456162] Key type lgssc registered [ 3298.813679] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3301.323376] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3301.329530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3302.492507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3303.752256] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 3305.500418] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3307.706174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3307.810682] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3309.633927] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 3311.361554] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3312.934388] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:131 to 0x240000400:161) [ 3323.361870] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3323.362080] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3323.366590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3327.279088] LustreError: 218043:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 3327.311028] Lustre: server umount lustre-OST0000 complete [ 3328.573894] LustreError: 218245:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 3328.575715] LustreError: 218245:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3328.632156] Lustre: server umount lustre-MDT0000 complete [ 3332.013267] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing unload_modules_local [ 3332.991485] Key type lgssc unregistered [ 3333.118345] LNet: 218777:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3333.122760] LNetError: 218777:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3333.132270] LNet: Removed LNI 192.168.203.111@tcp [ 3333.463808] Key type .llcrypt unregistered [ 3333.464939] Key type ._llcrypt unregistered [ 3336.695449] Key type ._llcrypt registered [ 3336.696932] Key type .llcrypt registered [ 3336.735839] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 3341.141646] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_hostid [ 3346.741660] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 3347.193311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3347.201837] alg: No test for adler32 (adler32-zlib) [ 3348.049917] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 3348.130362] LNet: Added LNI 192.168.203.111@tcp [8/256/0/180] [ 3349.712113] Key type lgssc registered [ 3350.050481] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3352.796747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3354.705019] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3356.753608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3358.972444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3358.995310] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3359.000060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3360.080794] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3360.095225] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3360.138494] Lustre: lustre-MDT0000: new disk, initializing [ 3360.161402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3360.167313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3361.376674] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 3364.366626] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3366.556761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3366.580181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3366.672940] Lustre: lustre-OST0000: new disk, initializing [ 3366.675876] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3366.677370] Lustre: Skipped 1 previous similar message [ 3366.701289] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3368.504120] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 3369.788863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3369.791885] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3369.799202] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3371.527110] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3372.832224] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3372.909115] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3375.073749] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3375.074712] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3375.082720] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3379.565089] LustreError: 223680:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 3379.597031] Lustre: server umount lustre-OST0000 complete [ 3380.842966] LustreError: 223881:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 3380.845327] LustreError: 223881:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3380.895896] Lustre: server umount lustre-MDT0000 complete [ 3383.454597] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 10:10:40 (1773238240) [ 3385.616798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3385.784728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3387.026214] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 3388.747172] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3390.923554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3391.030562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3392.798211] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 3394.467411] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3406.306384] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3406.306503] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3406.315430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3409.518116] LustreError: 225995:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 3409.520229] LustreError: 225995:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3409.552557] Lustre: server umount lustre-OST0000 complete [ 3410.864576] Lustre: server umount lustre-MDT0000 complete [ 3414.313710] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing unload_modules_local [ 3415.302208] Key type lgssc unregistered [ 3415.429349] LNet: 226730:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3415.431993] LNetError: 226730:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3415.438307] LNet: Removed LNI 192.168.203.111@tcp [ 3415.729959] Key type .llcrypt unregistered [ 3415.731617] Key type ._llcrypt unregistered [ 3423.098696] Key type ._llcrypt registered [ 3423.099747] Key type .llcrypt registered [ 3423.140164] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_hostid [ 3428.907325] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 3429.235981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3429.275142] alg: No test for adler32 (adler32-zlib) [ 3430.137054] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 3430.223664] LNet: Added LNI 192.168.203.111@tcp [8/256/0/180] [ 3431.808185] Key type lgssc registered [ 3432.167656] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3434.993340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3436.955620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3439.036521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3440.666390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3442.343625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3444.526674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3444.547609] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3444.551687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3445.630617] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3445.641680] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3445.681221] Lustre: lustre-MDT0000: new disk, initializing [ 3445.704568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3445.712510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3446.975614] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 3450.000323] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3452.312524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3452.339718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3452.429834] Lustre: lustre-OST2710: new disk, initializing [ 3452.432720] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 3452.435741] Lustre: Skipped 1 previous similar message [ 3452.460342] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 3454.015782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:2710:ost [ 3454.019275] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:2710:ost] [ 3454.028449] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x240000400 [ 3454.315215] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 3457.340588] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 3459.589911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3459.614609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3459.663875] Lustre: lustre-OST03e8: new disk, initializing [ 3459.665575] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 3459.689422] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 3461.524647] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 3463.275559] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 3464.197284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:3e8:ost [ 3464.200973] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:3e8:ost] [ 3464.209625] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x280000400 [ 3465.779486] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 3465.848440] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3467.086168] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 3467.164223] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3469.281753] Lustre: lustre-MDT0000-lwp-OST03e8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3469.282247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3469.286068] Lustre: Skipped 1 previous similar message [ 3470.208411] LustreError: 233064:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 3470.264294] Lustre: server umount lustre-MDT0000 complete [ 3471.502154] LustreError: 230080:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773238329 with bad export cookie 16288670254004533319 [ 3471.504361] LustreError: MGC192.168.203.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3471.505211] LustreError: 230080:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3473.583103] LustreError: 233267:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 3473.584878] LustreError: 233267:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3473.599755] Lustre: server umount lustre-OST2710 complete [ 3474.822848] LustreError: 233467:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3474.824958] LustreError: 233467:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3474.866277] Lustre: server umount lustre-OST03e8 complete [ 3479.667315] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_hostid [ 3482.539395] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 3485.747168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3487.740768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3489.819065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3492.077924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3492.097509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3492.176048] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3492.186733] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3492.222934] Lustre: lustre-MDT0000: new disk, initializing [ 3492.245787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3492.251667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3493.541247] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 3496.554073] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3498.807969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3498.830885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3498.920249] Lustre: lustre-OST0000: new disk, initializing [ 3498.923388] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3498.926404] Lustre: Skipped 1 previous similar message [ 3498.948729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3499.585295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3499.588272] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3499.598694] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3500.813662] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 3503.875931] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3505.179425] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3505.252301] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3509.729786] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3509.730186] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3509.736909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3509.739096] Lustre: Skipped 1 previous similar message [ 3511.918142] LustreError: 237659:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 3511.920141] LustreError: 237659:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3511.949099] Lustre: server umount lustre-OST0000 complete [ 3513.246992] Lustre: server umount lustre-MDT0000 complete [ 3515.854740] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 10:12:53 (1773238373) [ 3516.317231] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 3516.841877] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 10:12:54 (1773238374) [ 3517.802509] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 3520.391222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3521.666140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3522.978798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3523.341539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3525.533538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3525.622707] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3525.632543] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3525.694156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3526.968841] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 3528.684609] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3530.863437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3530.889257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3530.954523] Lustre: Denying initial registration attempt from nid 192.168.203.111@tcp, specified as failover [ 3530.957659] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 3530.961305] LustreError: 240591:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -99 [ 3530.963384] LustreError: 240591:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 3530.965271] LustreError: 240591:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 3530.986343] Lustre: server umount lustre-OST0000 complete [ 3530.987606] LustreError: 240591:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 3533.167561] LustreError: 240958:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3533.169847] LustreError: 240958:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3533.219729] Lustre: server umount lustre-MDT0000 complete [ 3536.583873] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing unload_modules_local [ 3537.528625] Key type lgssc unregistered [ 3537.652351] LNet: 241492:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3537.655067] LNetError: 241492:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3537.664303] LNet: Removed LNI 192.168.203.111@tcp [ 3537.935811] Key type .llcrypt unregistered [ 3537.937097] Key type ._llcrypt unregistered [ 3541.207664] Key type ._llcrypt registered [ 3541.208795] Key type .llcrypt registered [ 3541.245228] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 3541.736808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3541.744399] alg: No test for adler32 (adler32-zlib) [ 3542.595789] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 3542.673393] LNet: Added LNI 192.168.203.111@tcp [8/256/0/180] [ 3544.248173] Key type lgssc registered [ 3544.603296] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3546.843739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3548.116787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3549.398133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3549.745377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3555.877462] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 3558.661379] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3558.667743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3559.755507] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3559.763054] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3559.824280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3561.084444] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 3562.841277] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3565.047228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3565.073963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3565.139428] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3565.192558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3566.979058] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 3568.716341] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3571.681561] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3571.682156] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3571.689142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3575.470589] LustreError: 245485:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 3575.501701] Lustre: server umount lustre-OST0000 complete [ 3576.768438] LustreError: 245687:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 3576.770719] LustreError: 245687:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3576.825890] Lustre: server umount lustre-MDT0000 complete [ 3580.288510] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing unload_modules_local [ 3581.271846] Key type lgssc unregistered [ 3581.400418] LNet: 246219:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3581.404051] LNetError: 246219:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3581.414299] LNet: Removed LNI 192.168.203.111@tcp [ 3581.694407] Key type .llcrypt unregistered [ 3581.695757] Key type ._llcrypt unregistered [ 3590.352070] Key type ._llcrypt registered [ 3590.353149] Key type .llcrypt registered [ 3590.387727] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 3590.680439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3590.703352] alg: No test for adler32 (adler32-zlib) [ 3591.584141] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 3591.674993] LNet: Added LNI 192.168.203.111@tcp [8/256/0/180] [ 3593.256174] Key type lgssc registered [ 3593.641707] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3596.245162] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3596.251894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3597.423437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3598.762543] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 3600.552353] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3602.735262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3602.842381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3604.653143] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 3606.339926] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3618.030459] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.11@tcp (stopping) [ 3618.273258] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3620.796354] LustreError: 249039:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 3620.842458] Lustre: server umount lustre-MDT0000 complete [ 3622.915218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3625.772682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3625.809771] LustreError: MGC192.168.203.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3625.944409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3625.960936] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:166 to 0x240000400:193) [ 3627.235558] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 3628.976437] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3638.746745] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 9 sec [ 3639.203805] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3641.312863] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3641.317684] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3641.324620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3641.326507] Lustre: Skipped 1 previous similar message [ 3644.385136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3646.068093] LustreError: 250502:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 3646.070146] LustreError: 250502:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3646.107108] Lustre: server umount lustre-OST0000 complete [ 3647.394203] LustreError: 250703:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 3647.396678] LustreError: 250703:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3647.453212] Lustre: server umount lustre-MDT0000 complete [ 3651.138315] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing unload_modules_local [ 3652.160041] Key type lgssc unregistered [ 3652.279586] LNet: 251234:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3652.284323] LNetError: 251234:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3652.295315] LNet: Removed LNI 192.168.203.111@tcp [ 3652.628589] Key type .llcrypt unregistered [ 3652.629728] Key type ._llcrypt unregistered [ 3661.425955] Key type ._llcrypt registered [ 3661.426976] Key type .llcrypt registered [ 3661.464845] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 3661.752967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3661.760844] alg: No test for adler32 (adler32-zlib) [ 3662.611932] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 3662.692677] LNet: Added LNI 192.168.203.111@tcp [8/256/0/180] [ 3664.272135] Key type lgssc registered [ 3664.613258] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3667.051057] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3667.057614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3668.211485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3669.465302] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 3671.185299] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3672.260528] LustreError: 253200:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 3672.315566] Lustre: server umount lustre-MDT0000 complete [ 3677.012989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3677.102936] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3677.113471] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3677.189494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3678.415638] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 3680.121712] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3682.620612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3682.685396] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 3682.688579] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 3682.691351] LustreError: 254826:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 3682.693500] LustreError: 254826:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 3682.695501] LustreError: 254826:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 3682.718132] Lustre: server umount lustre-OST0000 complete [ 3682.719769] LustreError: 254826:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3685.325414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3685.393232] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3685.449487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3687.225109] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 3688.838883] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3690.997956] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:166 to 0x240000400:225) [ 3691.316516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3691.335296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3691.356063] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3691.381362] Lustre: lustre-OST0001: new disk, initializing [ 3691.382965] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3691.405865] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3693.289398] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 3694.993930] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3695.936109] LustreError: 256927:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3695.938292] LustreError: 256927:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3695.951448] Lustre: server umount lustre-OST0001 complete [ 3701.217230] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3701.219524] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3701.224829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3703.213064] LustreError: 257127:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 3703.214965] LustreError: 257127:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3703.248201] Lustre: server umount lustre-OST0000 complete [ 3704.577987] Lustre: server umount lustre-MDT0000 complete [ 3708.207057] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing unload_modules_local [ 3709.175702] Key type lgssc unregistered [ 3709.300416] LNet: 257864:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3709.302995] LNetError: 257864:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3709.314252] LNet: Removed LNI 192.168.203.111@tcp [ 3709.597844] Key type .llcrypt unregistered [ 3709.598991] Key type ._llcrypt unregistered [ 3718.077907] Key type ._llcrypt registered [ 3718.078994] Key type .llcrypt registered [ 3718.115189] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_hostid [ 3724.279333] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 3724.618690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3724.627443] alg: No test for adler32 (adler32-zlib) [ 3725.497671] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 3725.588468] LNet: Added LNI 192.168.203.111@tcp [8/256/0/180] [ 3727.184194] Key type lgssc registered [ 3727.594298] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3730.474770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3732.370587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3734.382158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3736.569185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3736.594029] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3736.598324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3737.680300] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3737.691634] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3737.732241] Lustre: lustre-MDT0000: new disk, initializing [ 3737.758131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3737.766683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3739.007977] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 3741.953499] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3744.142350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3744.165477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3744.256347] Lustre: lustre-OST0000: new disk, initializing [ 3744.258616] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3744.260628] Lustre: Skipped 1 previous similar message [ 3744.283647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3746.052932] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 3749.042669] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3750.362277] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3751.424806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3751.428739] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3751.435884] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3752.481231] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 3756.514070] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3756.514090] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3756.517955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3759.150530] LustreError: 263016:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 3759.182149] Lustre: server umount lustre-OST0000 complete [ 3760.571914] LustreError: 263218:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 3760.574924] LustreError: 263218:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3760.629350] Lustre: server umount lustre-MDT0000 complete [ 3763.231956] Lustre: DEBUG MARKER: == conf-sanity test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 10:17:00 (1773238620) [ 3765.164973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3770.712164] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_hostid [ 3773.450950] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 3776.748285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3778.661485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3780.740670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3783.013042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3783.040135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3783.128931] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3783.139996] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3783.180337] Lustre: lustre-MDT0000: new disk, initializing [ 3783.206637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3783.214483] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3784.505485] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 3787.559383] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3789.729324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3789.751964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3789.839862] Lustre: lustre-OST0000: new disk, initializing [ 3789.841753] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3789.843377] Lustre: Skipped 1 previous similar message [ 3789.866447] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3791.650378] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 3791.872109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3791.876353] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3791.883883] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3794.650847] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3795.931131] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3795.998142] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3796.961555] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3796.961589] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3796.966242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3802.081265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3802.671069] LustreError: 268031:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 3802.673053] LustreError: 268031:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3802.706202] Lustre: server umount lustre-OST0000 complete [ 3803.978457] Lustre: server umount lustre-MDT0000 complete [ 3806.624760] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 10:17:43 (1773238663) [ 3808.984523] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 10:17:46 (1773238666) [ 3811.699936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3811.858957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3813.101209] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 3814.774548] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3816.940066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3818.979001] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 3820.695269] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3830.069138] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 3830.618559] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 3831.178721] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 3831.699035] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 3832.335562] Lustre: Failing over lustre-MDT0000 [ 3832.510540] LustreError: 270885:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 3832.513388] LustreError: 270885:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3832.565686] Lustre: server umount lustre-MDT0000 complete [ 3845.388916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3845.433043] LustreError: MGC192.168.203.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3845.553487] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3845.577364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3845.579915] Lustre: Skipped 1 previous similar message [ 3845.597342] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3846.866974] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 3847.410726] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3847.418951] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 3847.435370] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:35 to 0x240000400:65) [ 3849.185399] Lustre: 259661:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773238690/real 1773238690] req@000000006e44ba98 x1859375420442240/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773238706 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3849.894082] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3850.410383] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3850.723942] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3852.245589] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 3852.901222] Lustre: Failing over lustre-MDT0000 [ 3853.030120] LustreError: 271970:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 3853.032160] LustreError: 271970:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3853.077859] Lustre: server umount lustre-MDT0000 complete [ 3853.920137] Lustre: 259659:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773238695/real 1773238695] req@000000001b02165d x1859375420442496/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773238711 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3858.912151] Lustre: 259661:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773238700/real 1773238700] req@000000006c644d68 x1859375420442752/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773238716 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3865.842030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3865.877291] LustreError: MGC192.168.203.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3865.880369] LustreError: 272392:0:(ldlm_resource.c:981:ldlm_resource_complain()) MGC192.168.203.111@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000f8081626) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 3865.885066] LustreError: 269237:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3865.991564] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3866.011814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3866.027761] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3867.234968] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 3867.891587] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3867.902515] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 3867.920052] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:35 to 0x240000400:97) [ 3870.296675] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3870.812760] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3871.204393] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3872.224128] Lustre: 259661:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773238713/real 1773238713] req@0000000000718977 x1859375420448256/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773238729 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3876.321563] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3876.327951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3877.088077] Lustre: 259659:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773238718/real 1773238718] req@000000000ee4192f x1859375420448512/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773238734 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3879.162228] LustreError: 273008:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 3879.163890] LustreError: 273008:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3879.205405] Lustre: server umount lustre-MDT0000 complete [ 3880.429174] LustreError: 269226:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773238738 with bad export cookie 17492578057479545449 [ 3880.431538] LustreError: MGC192.168.203.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3880.432811] LustreError: 269226:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3880.493181] Lustre: server umount lustre-OST0000 complete [ 3884.795725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3886.248199] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 3887.946750] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3890.165644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3892.138309] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 3893.895927] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3894.373492] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:35 to 0x240000400:129) [ 3904.993880] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3904.994457] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3905.002054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3906.158163] LustreError: 275383:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 3906.160275] LustreError: 275383:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3906.191937] Lustre: server umount lustre-OST0000 complete [ 3907.501493] Lustre: server umount lustre-MDT0000 complete [ 3910.988077] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing unload_modules_local [ 3911.956327] Key type lgssc unregistered [ 3912.095381] LNet: 276119:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3912.098159] LNetError: 276119:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3912.104420] LNet: Removed LNI 192.168.203.111@tcp [ 3912.390300] Key type .llcrypt unregistered [ 3912.391283] Key type ._llcrypt unregistered [ 3921.658223] Key type ._llcrypt registered [ 3921.659261] Key type .llcrypt registered [ 3921.696664] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 3922.032980] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3922.080901] alg: No test for adler32 (adler32-zlib) [ 3922.928556] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 3923.009496] LNet: Added LNI 192.168.203.111@tcp [8/256/0/180] [ 3924.584124] Key type lgssc registered [ 3924.929778] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3927.473450] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3927.478790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3928.629618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3929.894787] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 3931.630746] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3933.778155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3933.881188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3935.700721] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 3937.379724] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3938.983679] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:35 to 0x240000400:161) [ 3949.537467] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3949.545612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3953.275194] LustreError: 279037:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 3953.323498] Lustre: server umount lustre-MDT0000 complete [ 3954.578295] LustreError: 277570:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773238812 with bad export cookie 660267912567017329 [ 3954.580032] LustreError: MGC192.168.203.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3954.582110] LustreError: 277570:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3954.604140] LustreError: 279239:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3954.605949] LustreError: 279239:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3954.639618] Lustre: server umount lustre-OST0000 complete [ 3958.746621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3958.918657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3960.125750] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 3961.763608] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3963.944866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3964.041904] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3965.907849] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 3967.670967] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3969.190322] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:163 to 0x240000400:193) [ 3979.744934] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3979.749285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3984.865244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3985.469123] LustreError: 281515:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 3985.471564] LustreError: 281515:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3985.518476] Lustre: server umount lustre-MDT0000 complete [ 3986.734612] LustreError: 279990:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773238844 with bad export cookie 660267912567018113 [ 3986.736654] LustreError: MGC192.168.203.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3986.738570] LustreError: 279990:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3986.795349] Lustre: server umount lustre-OST0000 complete [ 3991.880870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3992.155889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3993.326679] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 3995.733499] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3998.043246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3999.968547] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 4001.732100] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4002.277798] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:163 to 0x240000400:225) [ 4012.513698] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4012.514337] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4012.520641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4014.191765] LustreError: 283897:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 4014.193705] LustreError: 283897:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4014.230700] Lustre: server umount lustre-OST0000 complete [ 4015.587097] Lustre: server umount lustre-MDT0000 complete [ 4019.009420] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing unload_modules_local [ 4020.000276] Key type lgssc unregistered [ 4020.125356] LNet: 284631:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4020.128613] LNetError: 284631:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4020.135335] LNet: Removed LNI 192.168.203.111@tcp [ 4020.422118] Key type .llcrypt unregistered [ 4020.422938] Key type ._llcrypt unregistered [ 4029.461071] Key type ._llcrypt registered [ 4029.462088] Key type .llcrypt registered [ 4029.497251] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 4029.815096] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4029.871277] alg: No test for adler32 (adler32-zlib) [ 4030.716462] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 4030.792804] LNet: Added LNI 192.168.203.111@tcp [8/256/0/180] [ 4032.368144] Key type lgssc registered [ 4032.717584] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4035.222166] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4035.227334] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4035.229689] LustreError: 286027:0:(osd_handler.c:8572:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 4035.238226] LustreError: 286027:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 4035.241120] LustreError: 286027:0:(obd_mount.c:193:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 4035.244127] LustreError: 286027:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 4035.247562] LustreError: 286027:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 4037.667414] LDISKFS-fs (dm-1): mounted filesystem without journal. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4037.669592] LustreError: 286425:0:(osd_handler.c:8572:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 4037.677859] LustreError: 286425:0:(obd_config.c:783:class_setup()) setup lustre-OST0000-osd failed (-22) [ 4037.679687] LustreError: 286425:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 4037.681617] LustreError: 286425:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 4037.683732] LustreError: 286425:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 4042.939477] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing unload_modules_local [ 4044.003527] Key type lgssc unregistered [ 4044.133324] LNet: 287255:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4044.137499] LNetError: 287255:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4044.145287] LNet: Removed LNI 192.168.203.111@tcp [ 4044.442426] Key type .llcrypt unregistered [ 4044.443569] Key type ._llcrypt unregistered [ 4049.153539] Key type ._llcrypt registered [ 4049.154624] Key type .llcrypt registered [ 4049.190433] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_hostid [ 4055.038527] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 4055.445601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4055.457651] alg: No test for adler32 (adler32-zlib) [ 4056.301733] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 4056.375586] LNet: Added LNI 192.168.203.111@tcp [8/256/0/180] [ 4057.952136] Key type lgssc registered [ 4058.292387] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4061.071790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4062.978803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4065.014384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4067.239714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4067.258573] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4067.263684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4068.348714] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4068.364581] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4068.408262] Lustre: lustre-MDT0000: new disk, initializing [ 4068.432270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4068.438380] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4069.679064] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 4072.708312] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4074.953089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4074.988604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4075.112915] Lustre: lustre-OST0000: new disk, initializing [ 4075.114760] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4075.116420] Lustre: Skipped 1 previous similar message [ 4075.139984] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4077.111720] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 4077.909636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 4077.915244] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 4077.929205] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 4080.293195] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4081.633583] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4081.724280] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4083.171078] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4083.171390] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4083.179187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4088.289608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4088.430114] LustreError: 291766:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 4088.464218] Lustre: server umount lustre-OST0000 complete [ 4089.775061] LustreError: 291968:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 4089.777727] LustreError: 291968:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 4089.833568] Lustre: server umount lustre-MDT0000 complete [ 4092.419897] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 4092.939160] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 10:22:30 (1773238950) [ 4095.058873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4095.236639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4096.590814] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 4098.341556] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4100.517094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4100.632168] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4102.414887] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 4104.097343] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4106.429846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4106.461096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4106.519043] Lustre: lustre-OST0001: new disk, initializing [ 4106.521409] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4106.551238] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4108.587884] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 4109.783611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 4109.789227] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 4109.799762] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 4111.753439] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4114.916146] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4114.917492] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4114.922570] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4118.256030] Lustre: lustre-OST0001: Not available for connect from 192.168.203.11@tcp (stopping) [ 4119.725081] LustreError: 295080:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4119.727121] LustreError: 295080:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4119.742718] Lustre: server umount lustre-OST0001 complete [ 4125.156557] LustreError: 294835:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4125.162885] LustreError: 294835:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4130.276228] LustreError: 293488:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4135.394017] LustreError: 293488:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4135.394029] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4135.394722] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4135.395214] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4140.513891] LustreError: 293489:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4145.633380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4145.634043] LustreError: 294835:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4145.636756] Lustre: Skipped 1 previous similar message [ 4146.144131] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4146.160072] LustreError: 295304:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 4146.163657] LustreError: 295304:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 4146.195957] Lustre: server umount lustre-OST0000 complete [ 4153.599972] LustreError: 295506:0:(obd_class.h:479:obd_check_dev()) Device 17 not setup [ 4153.603932] LustreError: 295506:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 4153.656884] Lustre: server umount lustre-MDT0000 complete [ 4157.151166] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing unload_modules_local [ 4158.170568] Key type lgssc unregistered [ 4158.291395] LNet: 296037:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4158.293743] LNetError: 296037:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4158.301276] LNet: Removed LNI 192.168.203.111@tcp [ 4158.597199] Key type .llcrypt unregistered [ 4158.598386] Key type ._llcrypt unregistered [ 4167.365498] Key type ._llcrypt registered [ 4167.366771] Key type .llcrypt registered [ 4167.403104] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_hostid [ 4173.388508] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 4173.849933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4173.862504] alg: No test for adler32 (adler32-zlib) [ 4174.706948] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 4174.793253] LNet: Added LNI 192.168.203.111@tcp [8/256/0/180] [ 4176.376125] Key type lgssc registered [ 4176.742545] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4179.483653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4181.432152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4183.404404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4185.591962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4185.613795] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4185.617856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4186.698550] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4186.710432] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4186.747817] Lustre: lustre-MDT0000: new disk, initializing [ 4186.771254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4186.777164] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4188.024462] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 4190.984383] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4193.152030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4193.174479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4193.266710] Lustre: lustre-OST0000: new disk, initializing [ 4193.269074] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4193.272483] Lustre: Skipped 1 previous similar message [ 4193.295308] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4195.100580] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 4198.181719] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4199.557700] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4201.474715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 4201.479073] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 4201.487814] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 4201.696732] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 4206.561979] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4206.561991] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4206.565897] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4208.429092] LustreError: 301174:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 4208.466491] Lustre: server umount lustre-OST0000 complete [ 4209.693488] LustreError: 301378:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 4209.695268] LustreError: 301378:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 4209.746309] Lustre: server umount lustre-MDT0000 complete [ 4212.324956] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 10:24:29 (1773239069) [ 4214.491317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4217.174238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4217.320286] Lustre: lustre-MDT0000: new disk, initializing [ 4217.357161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4218.633757] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 4220.354637] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4221.027188] LustreError: 303040:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 4221.028812] LustreError: 303040:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4221.073525] Lustre: server umount lustre-MDT0000 complete [ 4224.059476] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 10:24:41 (1773239081) [ 4226.216840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4226.383481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4227.604020] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 4229.498802] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4231.744337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4231.857306] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4233.770642] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 4235.485790] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4244.330081] Lustre: Permanently deactivating lustre-OST0000 [ 4244.334411] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 4251.630175] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4252.695956] LustreError: 305695:0:(mgs_llog.c:1938:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 4252.701118] LustreError: 305695:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [ 4252.704081] LustreError: 305695:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 4253.305769] LustreError: 305743:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [ 4253.308241] LustreError: 305743:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 4260.014328] LustreError: 305840:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 4260.016468] LustreError: 305840:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4260.049167] Lustre: server umount lustre-OST0000 complete [ 4260.952674] LustreError: 305992:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [ 4260.954968] LustreError: 305992:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 4272.099891] LustreError: 306089:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 4272.102889] LustreError: 306089:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 4272.152565] Lustre: server umount lustre-MDT0000 complete [ 4274.927314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4276.199179] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 4277.062784] LustreError: 306931:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 4279.035525] Lustre: 307269:0:(mgs_llog.c:1699:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.111@tcp [ 4279.689964] Lustre: server umount MGS complete [ 4282.395359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4282.542494] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4282.568615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4283.803747] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 4285.510566] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4286.114620] Lustre: Permanently reactivating lustre-OST0000 [ 4286.118355] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 4286.122744] Lustre: Skipped 1 previous similar message [ 4291.116781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4292.960490] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 4294.634434] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4307.937703] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4307.937911] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4307.943534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4308.334434] LustreError: 309556:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 4308.336705] LustreError: 309556:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4308.365875] Lustre: server umount lustre-OST0000 complete [ 4311.734383] Lustre: server umount lustre-MDT0000 complete [ 4315.271213] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing unload_modules_local [ 4316.197292] Key type lgssc unregistered [ 4316.316354] LNet: 310288:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4316.319955] LNetError: 310288:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4316.332291] LNet: Removed LNI 192.168.203.111@tcp [ 4316.613984] Key type .llcrypt unregistered [ 4316.615083] Key type ._llcrypt unregistered [ 4321.350773] Key type ._llcrypt registered [ 4321.351626] Key type .llcrypt registered [ 4321.386710] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_hostid [ 4327.201142] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 4327.516815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4327.530307] alg: No test for adler32 (adler32-zlib) [ 4328.384073] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 4328.465022] LNet: Added LNI 192.168.203.111@tcp [8/256/0/180] [ 4330.048158] Key type lgssc registered [ 4330.374567] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4333.033734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4335.017415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4336.960125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4339.194737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4339.219719] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4339.224554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4340.301270] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4340.310826] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4340.353863] Lustre: lustre-MDT0000: new disk, initializing [ 4340.377201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4340.383732] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4341.571325] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 4344.492090] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4346.597658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4346.620187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4346.703413] Lustre: lustre-OST0000: new disk, initializing [ 4346.704965] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4346.706415] Lustre: Skipped 1 previous similar message [ 4346.727697] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4348.495274] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 4348.605802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 4348.610250] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 4348.617421] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 4351.511080] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4352.804751] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4352.879843] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4354.018734] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4354.019010] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4354.029863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4359.137674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4359.535067] LustreError: 314799:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 4359.573844] Lustre: server umount lustre-OST0000 complete [ 4360.812407] LustreError: 315001:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 4360.814327] LustreError: 315001:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 4360.868858] Lustre: server umount lustre-MDT0000 complete [ 4363.520609] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 4364.038348] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 10:27:01 (1773239221) [ 4366.232847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4366.393392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4367.611857] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 4369.272611] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4371.405283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4371.508451] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4373.267112] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 4374.960461] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4375.578219] Lustre: ctl-lustre-MDT0000: [0x0000000280000400-0x00000002c0000400]:0:mdt sequences allocated: rc = 0 [ 4382.503254] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:0:mdt [ 4386.785897] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4386.785909] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4386.786680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4389.805766] LustreError: 317359:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 4389.808597] LustreError: 317359:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4389.839316] Lustre: server umount lustre-OST0000 complete [ 4391.119213] Lustre: server umount lustre-MDT0000 complete [ 4394.556933] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing unload_modules_local [ 4395.533756] Key type lgssc unregistered [ 4395.661349] LNet: 318091:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4395.663826] LNetError: 318091:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4395.675285] LNet: Removed LNI 192.168.203.111@tcp [ 4395.991640] Key type .llcrypt unregistered [ 4395.992955] Key type ._llcrypt unregistered [ 4412.488490] Key type ._llcrypt registered [ 4412.489397] Key type .llcrypt registered [ 4412.523491] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing unload_modules_local [ 4412.613265] Key type .llcrypt unregistered [ 4412.614222] Key type ._llcrypt unregistered [ 4418.524389] Key type ._llcrypt registered [ 4418.525368] Key type .llcrypt registered [ 4418.559480] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 4418.945753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4418.964894] alg: No test for adler32 (adler32-zlib) [ 4419.813479] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 4419.894947] LNet: Added LNI 192.168.203.111@tcp [8/256/0/180] [ 4421.472146] Key type lgssc registered [ 4421.826702] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4424.182559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4426.031598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4428.072278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4428.090087] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4428.094299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4429.172333] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4429.182253] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4429.216738] Lustre: lustre-MDT0000: new disk, initializing [ 4429.239300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4429.245114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4430.523748] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 4433.532400] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4435.616956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4435.641707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4435.734485] Lustre: lustre-OST0000: new disk, initializing [ 4435.736298] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4435.738458] Lustre: Skipped 1 previous similar message [ 4435.763089] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4436.799748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 4436.804471] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 4436.813622] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 4437.581336] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 4440.535242] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4447.201297] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4447.205107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4448.844981] LustreError: 323094:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 4448.907756] Lustre: server umount lustre-MDT0000 complete [ 4450.143637] LustreError: 321509:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773239307 with bad export cookie 5947374297975635458 [ 4450.147042] LustreError: MGC192.168.203.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4450.149930] LustreError: 321509:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4450.179724] LustreError: 323296:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4450.182202] LustreError: 323296:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4450.224910] Lustre: server umount lustre-OST0000 complete [ 4453.391257] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 10:28:30 (1773239310) [ 4453.771365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4456.089031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4456.243665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4457.437754] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 4459.264233] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4461.654824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4461.686939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4461.773064] Lustre: Found index 0 for lustre-OST0000, updating log [ 4461.856374] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4463.725180] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 4465.426250] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4466.981336] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5 to 0x240000400:33) [ 4472.290652] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4472.291178] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4472.297473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4473.313536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4474.031123] LustreError: 325577:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 4474.033349] LustreError: 325577:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4474.066679] Lustre: server umount lustre-OST0000 complete [ 4475.072544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4477.885542] LustreError: 325832:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 4477.887454] LustreError: 325832:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 4477.935696] Lustre: server umount lustre-MDT0000 complete [ 4480.727715] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 10:28:57 (1773239337) [ 4481.097528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4481.428308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4481.794112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4482.135054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4482.466189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4482.825499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4483.186314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4483.560059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4483.960193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4484.315226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4484.675278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4485.023908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4485.356620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4485.695943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4486.045813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4486.382160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4486.730847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4487.070284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4487.382162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4487.692853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4488.021178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4488.356218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4488.685058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4488.999083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4489.324150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4489.667727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4490.013286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4490.357985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4490.697778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4491.031964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4491.367220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4491.725751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4492.118822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4492.475367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4492.830757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4493.173240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4493.501462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4493.835177] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4494.163224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4494.481112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4494.809754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4495.128130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4495.438016] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4495.768193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4496.087172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4496.433929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4496.761501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4497.104437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4497.430686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4497.760195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4498.088310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4500.579289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4500.748478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4502.016827] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 4503.856509] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4506.762722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4506.813211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4506.927610] Lustre: Found index 0 for lustre-OST0000, updating log [ 4508.901180] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 4510.969778] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4511.141939] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5 to 0x240000400:65) [ 4516.321761] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4516.321988] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4516.329295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4517.745559] LustreError: 330729:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 4517.747251] LustreError: 330729:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4517.781634] Lustre: server umount lustre-OST0000 complete [ 4519.145515] Lustre: server umount lustre-MDT0000 complete [ 4520.081589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4524.514978] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_hostid [ 4527.811595] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 4531.186928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4533.136272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4535.293454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4537.426427] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 10:29:54 (1773239394) [ 4598.330116] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing unload_modules_local [ 4599.484274] Key type lgssc unregistered [ 4599.615439] LNet: 333950:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4599.618107] LNetError: 333950:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4724.583565] LNet: Removed LNI 192.168.203.111@tcp [ 4724.914240] Key type .llcrypt unregistered [ 4724.915659] Key type ._llcrypt unregistered [ 4731.807400] Key type ._llcrypt registered [ 4731.808821] Key type .llcrypt registered [ 4731.850574] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 4732.340297] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4732.365378] alg: No test for adler32 (adler32-zlib) [ 4733.274729] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 4733.392595] LNet: Added LNI 192.168.203.111@tcp [8/256/0/180] [ 4735.008213] Key type lgssc registered [ 4735.446550] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4736.918241] LNet: Added LNI 192.168.203.111@tcp100 [8/256/0/180] [ 4737.222859] LNet: Added LNI 192.168.203.111@tcp101 [8/256/0/180] [ 4737.541614] LNet: Added LNI 192.168.203.111@tcp102 [8/256/0/180] [ 4737.850021] LNet: Added LNI 192.168.203.111@tcp103 [8/256/0/180] [ 4738.163310] LNet: Added LNI 192.168.203.111@tcp104 [8/256/0/180] [ 4738.516454] LNet: Added LNI 192.168.203.111@tcp105 [8/256/0/180] [ 4738.930447] LNet: Added LNI 192.168.203.111@tcp106 [8/256/0/180] [ 4739.357489] LNet: Added LNI 192.168.203.111@tcp107 [8/256/0/180] [ 4739.762199] LNet: Added LNI 192.168.203.111@tcp108 [8/256/0/180] [ 4740.138446] LNet: Added LNI 192.168.203.111@tcp109 [8/256/0/180] [ 4740.485706] LNet: Added LNI 192.168.203.111@tcp110 [8/256/0/180] [ 4740.810347] LNet: Added LNI 192.168.203.111@tcp111 [8/256/0/180] [ 4741.121796] LNet: Added LNI 192.168.203.111@tcp112 [8/256/0/180] [ 4741.444506] LNet: Added LNI 192.168.203.111@tcp113 [8/256/0/180] [ 4741.816888] LNet: Added LNI 192.168.203.111@tcp114 [8/256/0/180] [ 4742.140683] LNet: Added LNI 192.168.203.111@tcp115 [8/256/0/180] [ 4742.510855] LNet: Added LNI 192.168.203.111@tcp116 [8/256/0/180] [ 4742.862815] LNet: Added LNI 192.168.203.111@tcp117 [8/256/0/180] [ 4743.230802] LNet: Added LNI 192.168.203.111@tcp118 [8/256/0/180] [ 4743.618967] LNet: Added LNI 192.168.203.111@tcp119 [8/256/0/180] [ 4744.039867] LNet: Added LNI 192.168.203.111@tcp120 [8/256/0/180] [ 4744.482455] LNet: Added LNI 192.168.203.111@tcp121 [8/256/0/180] [ 4744.929668] LNet: Added LNI 192.168.203.111@tcp122 [8/256/0/180] [ 4745.284222] LNet: Added LNI 192.168.203.111@tcp123 [8/256/0/180] [ 4745.631926] LNet: Added LNI 192.168.203.111@tcp124 [8/256/0/180] [ 4746.038353] LNet: Added LNI 192.168.203.111@tcp125 [8/256/0/180] [ 4746.347957] LNet: Added LNI 192.168.203.111@tcp126 [8/256/0/180] [ 4746.642214] LNet: Added LNI 192.168.203.111@tcp127 [8/256/0/180] [ 4746.940185] LNet: Added LNI 192.168.203.111@tcp128 [8/256/0/180] [ 4747.249584] LNet: Added LNI 192.168.203.111@tcp129 [8/256/0/180] [ 4747.557598] LNet: Added LNI 192.168.203.111@tcp130 [8/256/0/180] [ 4747.847190] LNet: Added LNI 192.168.203.111@tcp131 [8/256/0/180] [ 4748.154436] LNet: Added LNI 192.168.203.111@tcp132 [8/256/0/180] [ 4748.453660] LNet: Added LNI 192.168.203.111@tcp133 [8/256/0/180] [ 4748.768558] LNet: Added LNI 192.168.203.111@tcp134 [8/256/0/180] [ 4749.088763] LNet: Added LNI 192.168.203.111@tcp135 [8/256/0/180] [ 4751.361483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4751.385284] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4751.390316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4752.442147] Lustre: 336935:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.111@tcp: can't add NID 192.168.203.111@tcp130: rc = -75 [ 4752.489310] LNetError: 334398:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.111 [ 4752.501254] LNetError: Protocol error connecting to 192.168.203.111@tcp118 at host 192.168.203.111:988: is it running a compatible version of Lustre? [ 4752.503981] LNetError: Skipped 2 previous similar messages [ 4752.540868] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4752.551397] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4752.589969] Lustre: lustre-MDT0000: new disk, initializing [ 4752.621338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4752.627616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4752.865095] LNet: 2 local NIs in recovery (showing 2): 192.168.203.111@tcp123, 192.168.203.111@tcp121 [ 4753.570218] LNetError: 336967:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.111 [ 4753.573197] LNetError: 336967:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 4 previous similar messages [ 4753.576304] LNetError: Protocol error connecting to 192.168.203.111@tcp104 at host 192.168.203.111:988: is it running a compatible version of Lustre? [ 4753.579436] LNetError: Skipped 2 previous similar messages [ 4753.967834] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 4755.938455] LNetError: 334401:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.111 [ 4755.941359] LNetError: 334401:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 2 previous similar messages [ 4755.943525] LNetError: Protocol error connecting to 192.168.203.111@tcp121 at host 192.168.203.111:988: is it running a compatible version of Lustre? [ 4755.946605] LNetError: Skipped 2 previous similar messages [ 4756.997070] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4759.009698] LNetError: 334399:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.111 [ 4759.014162] LNetError: Protocol error connecting to 192.168.203.111@tcp121 at host 192.168.203.111:988: is it running a compatible version of Lustre? [ 4759.303759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4759.327196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4759.434791] Lustre: lustre-OST0000: new disk, initializing [ 4759.436737] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4759.438549] Lustre: Skipped 1 previous similar message [ 4759.470718] Lustre: 337853:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.111@tcp: can't add NID 192.168.203.111@tcp130: rc = -75 [ 4759.475174] Lustre: 337853:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 4759.484640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4760.395578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 4760.400955] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 4760.410829] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 4761.381638] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 4764.450509] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4767.201270] LNetError: 336969:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.111 [ 4767.204777] LNetError: Protocol error connecting to 192.168.203.111@tcp121 at host 192.168.203.111:988: is it running a compatible version of Lustre? [ 4768.814118] LustreError: 338681:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 4768.852574] Lustre: server umount lustre-OST0000 complete [ 4779.493026] LustreError: 338882:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 4779.495522] LustreError: 338882:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 4779.543533] Lustre: server umount lustre-MDT0000 complete [ 4782.976330] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing unload_modules_local [ 4783.586904] LNetError: 336958:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.111 [ 4783.594983] LNetError: Protocol error connecting to 192.168.203.111@tcp121 at host 192.168.203.111:988: is it running a compatible version of Lustre? [ 4783.956905] Key type lgssc unregistered [ 4784.110471] LNet: 339423:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4784.114591] LNetError: 339423:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4784.117541] LNet: Removed LNI 192.168.203.111@tcp [ 4785.120145] LNet: Removed LNI 192.168.203.111@tcp100 [ 4785.121292] LNetError: 339423:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4785.123014] LNetError: 339423:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 4785.124965] LNet: Removed LNI 192.168.203.111@tcp101 [ 4786.144212] LNet: Removed LNI 192.168.203.111@tcp102 [ 4786.145458] LNetError: 339423:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4786.147451] LNetError: 339423:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 4786.149415] LNet: Removed LNI 192.168.203.111@tcp103 [ 4787.168115] LNet: Removed LNI 192.168.203.111@tcp104 [ 4788.192121] LNet: Removed LNI 192.168.203.111@tcp105 [ 4788.194193] LNetError: 339423:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4788.197472] LNetError: 339423:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 4788.201708] LNet: Removed LNI 192.168.203.111@tcp106 [ 4789.216132] LNet: Removed LNI 192.168.203.111@tcp107 [ 4790.240153] LNet: Removed LNI 192.168.203.111@tcp108 [ 4791.264165] LNet: Removed LNI 192.168.203.111@tcp109 [ 4792.288147] LNet: Removed LNI 192.168.203.111@tcp110 [ 4792.290107] LNetError: 339423:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4792.293438] LNetError: 339423:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 4792.296990] LNet: Removed LNI 192.168.203.111@tcp111 [ 4793.312097] LNet: Removed LNI 192.168.203.111@tcp112 [ 4794.336141] LNet: Removed LNI 192.168.203.111@tcp113 [ 4795.360098] LNet: Removed LNI 192.168.203.111@tcp114 [ 4796.384143] LNet: Removed LNI 192.168.203.111@tcp115 [ 4797.408098] LNet: Removed LNI 192.168.203.111@tcp116 [ 4798.432124] LNet: Removed LNI 192.168.203.111@tcp117 [ 4799.456136] LNet: Removed LNI 192.168.203.111@tcp118 [ 4800.480142] LNet: Removed LNI 192.168.203.111@tcp119 [ 4800.482587] LNetError: 339423:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4800.485326] LNetError: 339423:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 4800.488113] LNet: Removed LNI 192.168.203.111@tcp120 [ 4800.489972] LNet: Removed LNI 192.168.203.111@tcp121 [ 4801.504121] LNet: Removed LNI 192.168.203.111@tcp122 [ 4802.528103] LNet: Removed LNI 192.168.203.111@tcp123 [ 4802.529253] LNet: Removed LNI 192.168.203.111@tcp124 [ 4802.530743] LNet: Removed LNI 192.168.203.111@tcp125 [ 4802.531985] LNet: Removed LNI 192.168.203.111@tcp126 [ 4802.533083] LNet: Removed LNI 192.168.203.111@tcp127 [ 4802.534128] LNet: Removed LNI 192.168.203.111@tcp128 [ 4802.535077] LNet: Removed LNI 192.168.203.111@tcp129 [ 4802.535989] LNet: Removed LNI 192.168.203.111@tcp130 [ 4802.536763] LNet: Removed LNI 192.168.203.111@tcp131 [ 4802.537585] LNet: Removed LNI 192.168.203.111@tcp132 [ 4802.538547] LNet: Removed LNI 192.168.203.111@tcp133 [ 4802.539529] LNet: Removed LNI 192.168.203.111@tcp134 [ 4802.546469] LNet: Removed LNI 192.168.203.111@tcp135 [ 4802.803254] Key type .llcrypt unregistered [ 4802.804297] Key type ._llcrypt unregistered [ 4808.992736] Key type ._llcrypt registered [ 4808.994354] Key type .llcrypt registered [ 4809.031094] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_hostid [ 4815.196353] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 4815.502702] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4815.520379] alg: No test for adler32 (adler32-zlib) [ 4816.373569] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 4816.456859] LNet: Added LNI 192.168.203.111@tcp [8/256/0/180] [ 4818.032112] Key type lgssc registered [ 4818.364908] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4821.334385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4823.239562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4825.209835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4827.395993] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 10:34:44 (1773239684) [ 4832.075263] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing unload_modules_local [ 4833.113317] Key type lgssc unregistered [ 4833.242291] LNet: 342870:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4833.244338] LNetError: 342870:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4833.253271] LNet: Removed LNI 192.168.203.111@tcp [ 4833.525527] Key type .llcrypt unregistered [ 4833.526710] Key type ._llcrypt unregistered [ 4839.643896] Key type ._llcrypt registered [ 4839.645227] Key type .llcrypt registered [ 4839.690129] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 4840.071373] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4840.090046] alg: No test for adler32 (adler32-zlib) [ 4840.951322] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 4841.035564] LNet: Added LNI 192.168.203.111@tcp [8/256/0/180] [ 4842.616234] Key type lgssc registered [ 4843.111857] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4846.196991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4846.220834] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4846.226802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4847.312292] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4847.323218] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4847.364750] Lustre: lustre-MDT0000: new disk, initializing [ 4847.389036] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4847.395230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4848.649850] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 4851.683592] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4853.922920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4853.952884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4854.054978] Lustre: lustre-OST0000: new disk, initializing [ 4854.057862] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4854.060635] Lustre: Skipped 1 previous similar message [ 4854.085283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4855.899106] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 4858.933139] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4860.041852] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing lnet_if_list [ 4861.564750] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 4861.575951] LNet: Added LNI 192.168.203.111@tcp42 [8/256/0/180] [ 4863.492892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 4863.497095] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 4863.506206] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 4868.578038] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4868.578122] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4868.582247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4869.806396] LustreError: 346555:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 4869.837212] Lustre: server umount lustre-OST0000 complete [ 4873.218568] LustreError: 346757:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 4873.220272] LustreError: 346757:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 4873.273242] Lustre: server umount lustre-MDT0000 complete [ 4876.645178] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing unload_modules_local [ 4877.641528] Key type lgssc unregistered [ 4877.797348] LNet: 347296:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4877.799827] LNetError: 347296:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4877.803089] LNet: Removed LNI 192.168.203.111@tcp [ 4877.810309] LNet: Removed LNI 192.168.203.111@tcp42 [ 4878.076051] Key type .llcrypt unregistered [ 4878.077045] Key type ._llcrypt unregistered [ 4885.057876] Key type ._llcrypt registered [ 4885.059381] Key type .llcrypt registered [ 4885.100263] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 4885.397963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4885.428994] alg: No test for adler32 (adler32-zlib) [ 4886.286724] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 4886.375413] LNet: Added LNI 192.168.203.111@tcp [8/256/0/180] [ 4887.960163] Key type lgssc registered [ 4888.312316] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4890.869693] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4890.877342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4892.041651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4893.332306] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 4895.003124] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4897.278726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4897.395495] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4899.259275] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 4901.100865] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4911.095438] loop0: detected capacity change from 0 to 204800000 [ 4911.155959] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4911.159412] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4911.163976] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4911.196564] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4913.164074] loop0: detected capacity change from 0 to 204800000 [ 4913.220379] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4913.223697] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4913.226810] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4913.281107] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4915.139474] loop0: detected capacity change from 0 to 204800000 [ 4916.959057] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4916.978196] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4917.021470] Lustre: fs15246-OST0000: new disk, initializing [ 4917.023532] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 4917.047115] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4918.853510] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 4922.529031] loop1: detected capacity change from 0 to 204800000 [ 4924.320588] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4924.349800] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4924.382675] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 4924.408877] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 4924.444844] Lustre: fs15246-MDT0000: new disk, initializing [ 4924.494662] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4924.505179] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4924.540716] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 4925.600684] Lustre: 352095:0:(client.c:1627:after_reply()) @@@ resending request on EINPROGRESS req@00000000f261f7f3 x1859376637821312/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1773239799 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 4925.766785] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 4933.612354] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 4934.688274] LustreError: 352095:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 4934.691683] LustreError: 352095:0:(fid_request.c:255:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [ 4934.694729] LustreError: 352095:0:(osp_precreate.c:459:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [ 4938.722818] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 4939.208803] LustreError: 353213:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4939.236359] Lustre: server umount fs15246-MDT0000 complete [ 4942.167786] LustreError: 353719:0:(obd_class.h:479:obd_check_dev()) Device 20 not setup [ 4942.169761] LustreError: 353719:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4942.184977] Lustre: server umount fs15246-OST0000 complete [ 4947.936908] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4947.940556] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4947.943807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4951.535276] LustreError: 354219:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 4951.537223] LustreError: 354219:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 4951.567352] Lustre: server umount lustre-OST0000 complete [ 4953.008318] Lustre: server umount lustre-MDT0000 complete [ 4956.385838] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing unload_modules_local [ 4957.360978] Key type lgssc unregistered [ 4957.488418] LNet: 354952:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4957.492181] LNetError: 354952:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4957.499284] LNet: Removed LNI 192.168.203.111@tcp [ 4957.800100] Key type .llcrypt unregistered [ 4957.801770] Key type ._llcrypt unregistered [ 4964.384104] Key type ._llcrypt registered [ 4964.386016] Key type .llcrypt registered [ 4964.430276] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 4964.810938] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4964.863207] alg: No test for adler32 (adler32-zlib) [ 4965.712241] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 4965.792946] LNet: Added LNI 192.168.203.111@tcp [8/256/0/180] [ 4967.376117] Key type lgssc registered [ 4967.711237] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4970.030485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4971.607710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4973.218318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4974.781963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4978.890187] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_hostid [ 4981.622588] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 4984.634435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4986.482694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4988.477885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4990.702156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4990.720534] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4990.726358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4991.810387] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4991.822887] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4991.864864] Lustre: lustre-MDT0000: new disk, initializing [ 4991.891122] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4991.898329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4993.169720] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 4996.181884] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4998.383679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4998.409462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4998.496051] Lustre: lustre-OST0000: new disk, initializing [ 4998.498077] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4998.500633] Lustre: Skipped 1 previous similar message [ 4998.523595] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4999.939385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 4999.942506] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 4999.952143] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 5000.314261] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_default_debug -1 all [ 5003.288848] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5004.581624] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5004.650598] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5005.281442] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5005.281529] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5005.285079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5010.401322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5011.374866] LustreError: 360756:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 5011.412019] Lustre: server umount lustre-OST0000 complete [ 5012.710980] LustreError: 360958:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 5012.713980] LustreError: 360958:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5012.770867] Lustre: server umount lustre-MDT0000 complete [ 5019.623261] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing set_hostid [ 5022.358616] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing load_modules_local [ 5025.570364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5027.591576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5029.746602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5030.318093] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 4834 sec ========== 10:38:07 (1773239887) [ 5030.873241] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 10:38:08 (1773239888) === [ 5031.917249] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 10:38:09 (1773239889) === [ 5041.879964] Lustre: DEBUG MARKER: oleg311-server.virtnet: executing unload_modules_local [ 5042.835449] Key type lgssc unregistered [ 5042.950309] LNet: 364987:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5042.952265] LNetError: 364987:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5042.963266] LNet: Removed LNI 192.168.203.111@tcp [ 5043.226359] Key type .llcrypt unregistered [ 5043.227301] Key type ._llcrypt unregistered