[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-7.9-debug (green@centos7-base) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Sat Mar 26 23:28:42 EDT 2022 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000013edfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] e820: last_pfn = 0x13ee00 max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: write-back [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF uncachable [ 0.000000] C0000-FFFFF write-protect [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 0000C0000000 mask 3FFFC0000000 uncachable [ 0.000000] 1 disabled [ 0.000000] 2 disabled [ 0.000000] 3 disabled [ 0.000000] 4 disabled [ 0.000000] 5 disabled [ 0.000000] 6 disabled [ 0.000000] 7 disabled [ 0.000000] PAT configuration [0-7]: WB WC UC- UC WB WP UC- UC [ 0.000000] e820: last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] mapped at [ffffffffff200b30] [ 0.000000] Base memory trampoline at [ffff880000099000] 99000 size 24576 [ 0.000000] Using GB pages for direct mapping [ 0.000000] BRK [0x023c1000, 0x023c1fff] PGTABLE [ 0.000000] BRK [0x023c2000, 0x023c2fff] PGTABLE [ 0.000000] BRK [0x023c3000, 0x023c3fff] PGTABLE [ 0.000000] BRK [0x023c4000, 0x023c4fff] PGTABLE [ 0.000000] BRK [0x023c5000, 0x023c5fff] PGTABLE [ 0.000000] BRK [0x023c6000, 0x023c6fff] PGTABLE [ 0.000000] RAMDISK: [mem 0xbc2e2000-0xbffbffff] [ 0.000000] Early table checksum verification disabled [ 0.000000] ACPI: RSDP 00000000000f5950 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 00000000bffe1bb7 00034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 00000000bffe1a53 00074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 00000000bffe0040 01A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 00000000bffe0000 00040 [ 0.000000] ACPI: APIC 00000000bffe1ac7 00090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 00000000bffe1b57 00038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 00000000bffe1b8f 00028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000013edfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x13e5e3000-0x13e609fff] [ 0.000000] Reserving 128MB of memory at 768MB for crashkernel (System RAM: 4077MB) [ 0.000000] kvm-clock: cpu 0, msr 1:3e592001, primary cpu clock [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 303070451 cycles [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal [mem 0x100000000-0x13edfffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009efff] [ 0.000000] node 0: [mem 0x00100000-0xbffcdfff] [ 0.000000] node 0: [mem 0x100000000-0x13edfffff] [ 0.000000] Initmem setup node 0 [mem 0x00001000-0x13edfffff] [ 0.000000] On node 0 totalpages: 1043820 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 21 pages reserved [ 0.000000] DMA zone: 3998 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 12224 pages used for memmap [ 0.000000] DMA32 zone: 782286 pages, LIFO batch:31 [ 0.000000] Normal zone: 4024 pages used for memmap [ 0.000000] Normal zone: 257536 pages, LIFO batch:31 [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ5 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] ACPI: IRQ10 used by override. [ 0.000000] ACPI: IRQ11 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] e820: [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 38 pages/cpu s115176 r8192 d32280 u524288 [ 0.000000] pcpu-alloc: s115176 r8192 d32280 u524288 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 0 1 2 3 [ 0.000000] KVM setup async PF for cpu 0 [ 0.000000] kvm-stealtime: cpu 0, msr 13e2135c0 [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes) [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 1027487 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] audit: disabled (until reboot) [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 3820268k/5224448k available (8172k kernel code, 1049168k absent, 355012k reserved, 5773k data, 2532k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=4. [ 0.000000] Offload RCU callbacks from all CPUs [ 0.000000] Offload RCU callbacks from CPUs: 0-3. [ 0.000000] NR_IRQS:327936 nr_irqs:456 0 [ 0.000000] Console: colour *CGA 80x25 [ 0.000000] console [ttyS1] enabled [ 0.000000] allocated 25165824 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ODEBUG: 73 of 73 active objects replaced [ 0.000000] hpet clockevent registered [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.425227] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.427923] pid_max: default: 32768 minimum: 301 [ 0.430197] Security Framework initialized [ 0.432119] SELinux: Initializing. [ 0.433488] SELinux: Starting in permissive mode [ 0.435086] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes) [ 0.439027] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.442338] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.444285] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.446946] Initializing cgroup subsys memory [ 0.448042] Initializing cgroup subsys devices [ 0.449387] Initializing cgroup subsys freezer [ 0.451631] Initializing cgroup subsys net_cls [ 0.453217] Initializing cgroup subsys blkio [ 0.455496] Initializing cgroup subsys perf_event [ 0.456830] Initializing cgroup subsys hugetlb [ 0.458489] Initializing cgroup subsys pids [ 0.459924] Initializing cgroup subsys net_prio [ 0.461740] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.464988] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.466956] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.468915] tlb_flushall_shift: 6 [ 0.470125] FEATURE SPEC_CTRL Present [ 0.471363] FEATURE IBPB_SUPPORT Present [ 0.472362] Spectre V2 : Enabling Indirect Branch Prediction Barrier [ 0.474373] Spectre V2 : Vulnerable [ 0.475634] Speculative Store Bypass: Vulnerable [ 0.478323] debug: unmapping init [mem 0xffffffff82019000-0xffffffff8201ffff] [ 0.485479] ACPI: Core revision 20130517 [ 0.487565] ACPI: All ACPI Tables successfully acquired [ 0.488753] ftrace: allocating 30294 entries in 119 pages [ 0.544755] Enabling x2apic [ 0.545923] Enabled x2apic [ 0.547254] Switched APIC routing to physical x2apic. [ 0.549183] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.550420] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (fam: 06, model: 3e, stepping: 04) [ 0.552118] TSC deadline timer enabled [ 0.552265] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.553724] ... version: 2 [ 0.554381] ... bit width: 48 [ 0.555109] ... generic registers: 4 [ 0.555769] ... value mask: 0000ffffffffffff [ 0.557489] ... max period: 00007fffffffffff [ 0.559435] ... fixed-purpose events: 3 [ 0.560877] ... event mask: 000000070000000f [ 0.562725] KVM setup paravirtual spinlock [ 0.568183] kvm-clock: cpu 1, msr 1:3e592041, secondary cpu clock [ 0.572501] KVM setup async PF for cpu 1 [ 0.573750] kvm-stealtime: cpu 1, msr 13e2935c0 [ 0.579098] kvm-clock: cpu 2, msr 1:3e592081, secondary cpu clock [ 0.582122] KVM setup async PF for cpu 2 [ 0.566213] smpboot: Booting Node 0, Processors #1 #2 #3 OK [ 0.582794] kvm-clock: cpu 3, msr 1:3e5920c1, secondary cpu clock [ 0.586529] kvm-stealtime: cpu 2, msr 13e3135c0 [ 0.587941] Brought up 4 CPUs [ 0.587948] KVM setup async PF for cpu 3 [ 0.587954] kvm-stealtime: cpu 3, msr 13e3935c0 [ 0.591046] smpboot: Max logical packages: 1 [ 0.592642] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.596842] devtmpfs: initialized [ 0.598162] x86/mm: Memory block size: 128MB [ 0.604035] EVM: security.selinux [ 0.605061] EVM: security.ima [ 0.605951] EVM: security.capability [ 0.608926] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.610326] NET: Registered protocol family 16 [ 0.612062] cpuidle: using governor haltpoll [ 0.613845] ACPI: bus type PCI registered [ 0.615124] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.616475] PCI: Using configuration type 1 for base access [ 0.618136] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.627395] ACPI: Added _OSI(Module Device) [ 0.628907] ACPI: Added _OSI(Processor Device) [ 0.630396] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.631894] ACPI: Added _OSI(Processor Aggregator Device) [ 0.633493] ACPI: Added _OSI(Linux-Dell-Video) [ 0.635908] ACPI: EC: Look up EC in DSDT [ 0.638657] ACPI: Interpreter enabled [ 0.639972] ACPI: (supports S0 S3 S4 S5) [ 0.641345] ACPI: Using IOAPIC for interrupt routing [ 0.642906] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.645274] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.651454] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.653555] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.655708] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.657567] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.662250] acpiphp: Slot [2] registered [ 0.663714] acpiphp: Slot [3] registered [ 0.665257] acpiphp: Slot [4] registered [ 0.666815] acpiphp: Slot [5] registered [ 0.668375] acpiphp: Slot [6] registered [ 0.669631] acpiphp: Slot [7] registered [ 0.670939] acpiphp: Slot [8] registered [ 0.672435] acpiphp: Slot [9] registered [ 0.673928] acpiphp: Slot [10] registered [ 0.675405] acpiphp: Slot [11] registered [ 0.676875] acpiphp: Slot [12] registered [ 0.678538] acpiphp: Slot [13] registered [ 0.680236] acpiphp: Slot [14] registered [ 0.682633] acpiphp: Slot [15] registered [ 0.684325] acpiphp: Slot [16] registered [ 0.686027] acpiphp: Slot [17] registered [ 0.687737] acpiphp: Slot [18] registered [ 0.689854] acpiphp: Slot [19] registered [ 0.691560] acpiphp: Slot [20] registered [ 0.693221] acpiphp: Slot [21] registered [ 0.694914] acpiphp: Slot [22] registered [ 0.696556] acpiphp: Slot [23] registered [ 0.698185] acpiphp: Slot [24] registered [ 0.699823] acpiphp: Slot [25] registered [ 0.701430] acpiphp: Slot [26] registered [ 0.703070] acpiphp: Slot [27] registered [ 0.704735] acpiphp: Slot [28] registered [ 0.706375] acpiphp: Slot [29] registered [ 0.708042] acpiphp: Slot [30] registered [ 0.709699] acpiphp: Slot [31] registered [ 0.711295] PCI host bridge to bus 0000:00 [ 0.712855] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.715390] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.717933] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.720672] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.723455] pci_bus 0000:00: root bus resource [mem 0x140000000-0x1bfffffff window] [ 0.726143] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.727841] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.728588] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.729505] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.735078] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.737277] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.740419] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.742042] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.743943] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.746016] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.746650] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.748904] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.751578] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.753386] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.764228] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.766650] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.767764] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.772867] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.777694] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.789033] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.806534] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.812048] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.821084] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.834880] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.843093] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.846259] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.849210] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.860750] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.871493] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.878181] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.887606] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.903598] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.913827] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.922287] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.927267] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.945360] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.955827] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.960543] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.967948] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.978424] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.983010] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 0.985065] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 0.988222] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 0.990343] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 0.991981] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 0.995263] vgaarb: loaded [ 0.996443] SCSI subsystem initialized [ 0.997417] ACPI: bus type USB registered [ 0.998347] usbcore: registered new interface driver usbfs [ 0.999770] usbcore: registered new interface driver hub [ 1.001152] usbcore: registered new device driver usb [ 1.003180] PCI: Using ACPI for IRQ routing [ 1.004632] PCI: pci_cache_line_size set to 64 bytes [ 1.005006] e820: reserve RAM buffer [mem 0x0009fc00-0x0009ffff] [ 1.005013] e820: reserve RAM buffer [mem 0xbffce000-0xbfffffff] [ 1.005015] e820: reserve RAM buffer [mem 0x13ee00000-0x13fffffff] [ 1.005950] NetLabel: Initializing [ 1.007171] NetLabel: domain hash size = 128 [ 1.008750] NetLabel: protocols = UNLABELED CIPSOv4 [ 1.010476] NetLabel: unlabeled traffic allowed by default [ 1.012721] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.014249] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.019308] amd_nb: Cannot enumerate AMD northbridges [ 1.021327] Switched to clocksource kvm-clock [ 1.041234] pnp: PnP ACPI init [ 1.042341] ACPI: bus type PNP registered [ 1.043586] pnp 00:00: Plug and Play ACPI device, IDs PNP0303 (active) [ 1.043652] pnp 00:01: Plug and Play ACPI device, IDs PNP0f13 (active) [ 1.043684] pnp 00:02: [dma 2] [ 1.043707] pnp 00:02: Plug and Play ACPI device, IDs PNP0700 (active) [ 1.043758] pnp 00:03: Plug and Play ACPI device, IDs PNP0501 (active) [ 1.043811] pnp 00:04: Plug and Play ACPI device, IDs PNP0501 (active) [ 1.043887] pnp 00:05: Plug and Play ACPI device, IDs PNP0b00 (active) [ 1.044197] pnp: PnP ACPI: found 6 devices [ 1.045444] ACPI: bus type PNP unregistered [ 1.056847] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.056853] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.056855] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.056857] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.056859] pci_bus 0000:00: resource 8 [mem 0x140000000-0x1bfffffff window] [ 1.056938] NET: Registered protocol family 2 [ 1.058563] TCP established hash table entries: 32768 (order: 6, 262144 bytes) [ 1.061355] TCP bind hash table entries: 32768 (order: 8, 1048576 bytes) [ 1.064155] TCP: Hash tables configured (established 32768 bind 32768) [ 1.066646] TCP: reno registered [ 1.068053] UDP hash table entries: 2048 (order: 5, 196608 bytes) [ 1.070560] UDP-Lite hash table entries: 2048 (order: 5, 196608 bytes) [ 1.073088] NET: Registered protocol family 1 [ 1.075032] RPC: Registered named UNIX socket transport module. [ 1.076196] RPC: Registered udp transport module. [ 1.077104] RPC: Registered tcp transport module. [ 1.078011] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.079194] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.080776] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.082578] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.084483] PCI: CLS 0 bytes, default 64 [ 1.084755] Unpacking initramfs... [ 2.453835] debug: unmapping init [mem 0xffff8800bc2e2000-0xffff8800bffbffff] [ 2.457886] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.460011] software IO TLB [mem 0xb82e2000-0xbc2e2000] (64MB) mapped at [ffff8800b82e2000-ffff8800bc2e1fff] [ 2.463189] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 10737418240 ms ovfl timer [ 2.465543] RAPL PMU: hw unit of domain pp0-core 2^-0 Joules [ 2.466913] RAPL PMU: hw unit of domain package 2^-0 Joules [ 2.467994] RAPL PMU: hw unit of domain dram 2^-0 Joules [ 2.473259] cryptomgr_test (52) used greatest stack depth: 14128 bytes left [ 2.474315] futex hash table entries: 1024 (order: 4, 65536 bytes) [ 2.474360] Initialise system trusted keyring [ 2.506541] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 2.508792] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 2.516571] zpool: loaded [ 2.517633] zbud: loaded [ 2.518954] VFS: Disk quotas dquot_6.6.0 [ 2.520452] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 2.524495] NFS: Registering the id_resolver key type [ 2.526217] Key type id_resolver registered [ 2.529934] Key type id_legacy registered [ 2.531458] nfs4filelayout_init: NFSv4 File Layout Driver Registering... [ 2.536890] Key type big_key registered [ 2.538963] SELinux: Registering netfilter hooks [ 2.544401] cryptomgr_test (58) used greatest stack depth: 13968 bytes left [ 2.550412] cryptomgr_test (60) used greatest stack depth: 13664 bytes left [ 2.558580] NET: Registered protocol family 38 [ 2.559948] Key type asymmetric registered [ 2.560975] Asymmetric key parser 'x509' registered [ 2.563583] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250) [ 2.568515] io scheduler noop registered [ 2.569661] io scheduler deadline registered (default) [ 2.571793] io scheduler cfq registered [ 2.572882] io scheduler mq-deadline registered [ 2.574127] io scheduler kyber registered [ 2.578529] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 2.580844] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 2.583221] intel_idle: does not run on family 6 model 62 [ 2.583385] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.585747] ACPI: Power Button [PWRF] [ 2.587405] GHES: HEST is not enabled! [ 2.646528] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 10 [ 2.707077] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 11 [ 2.827919] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 [ 2.889104] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 10 [ 3.018196] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.045686] 00:03: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A [ 3.075151] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 3.079029] Non-volatile memory driver v1.3 [ 3.080710] Linux agpgart interface v0.103 [ 3.082583] crash memory driver: version 1.1 [ 3.084676] nbd: registered device at major 43 [ 3.092781] virtio-pci 0000:00:05.0: irq 24 for MSI/MSI-X [ 3.092825] virtio-pci 0000:00:05.0: irq 25 for MSI/MSI-X [ 3.092879] virtio-pci 0000:00:05.0: irq 26 for MSI/MSI-X [ 3.092919] virtio-pci 0000:00:05.0: irq 27 for MSI/MSI-X [ 3.092957] virtio-pci 0000:00:05.0: irq 28 for MSI/MSI-X [ 3.101954] virtio_blk virtio1: [vda] 115728 512-byte logical blocks (59.2 MB/56.5 MiB) [ 3.110275] virtio-pci 0000:00:06.0: irq 29 for MSI/MSI-X [ 3.110353] virtio-pci 0000:00:06.0: irq 30 for MSI/MSI-X [ 3.110393] virtio-pci 0000:00:06.0: irq 31 for MSI/MSI-X [ 3.110434] virtio-pci 0000:00:06.0: irq 32 for MSI/MSI-X [ 3.110475] virtio-pci 0000:00:06.0: irq 33 for MSI/MSI-X [ 3.117111] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.132439] virtio-pci 0000:00:07.0: irq 34 for MSI/MSI-X [ 3.132488] virtio-pci 0000:00:07.0: irq 35 for MSI/MSI-X [ 3.132549] virtio-pci 0000:00:07.0: irq 36 for MSI/MSI-X [ 3.132585] virtio-pci 0000:00:07.0: irq 37 for MSI/MSI-X [ 3.132629] virtio-pci 0000:00:07.0: irq 38 for MSI/MSI-X [ 3.138608] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.146132] virtio-pci 0000:00:08.0: irq 39 for MSI/MSI-X [ 3.146178] virtio-pci 0000:00:08.0: irq 40 for MSI/MSI-X [ 3.146216] virtio-pci 0000:00:08.0: irq 41 for MSI/MSI-X [ 3.146262] virtio-pci 0000:00:08.0: irq 42 for MSI/MSI-X [ 3.146325] virtio-pci 0000:00:08.0: irq 43 for MSI/MSI-X [ 3.157816] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.165072] virtio-pci 0000:00:09.0: irq 44 for MSI/MSI-X [ 3.165113] virtio-pci 0000:00:09.0: irq 45 for MSI/MSI-X [ 3.165153] virtio-pci 0000:00:09.0: irq 46 for MSI/MSI-X [ 3.165200] virtio-pci 0000:00:09.0: irq 47 for MSI/MSI-X [ 3.165244] virtio-pci 0000:00:09.0: irq 48 for MSI/MSI-X [ 3.171361] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.181433] virtio-pci 0000:00:0a.0: irq 49 for MSI/MSI-X [ 3.181468] virtio-pci 0000:00:0a.0: irq 50 for MSI/MSI-X [ 3.181497] virtio-pci 0000:00:0a.0: irq 51 for MSI/MSI-X [ 3.181548] virtio-pci 0000:00:0a.0: irq 52 for MSI/MSI-X [ 3.181578] virtio-pci 0000:00:0a.0: irq 53 for MSI/MSI-X [ 3.187616] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.193441] rdac: device handler registered [ 3.195514] hp_sw: device handler registered [ 3.197042] emc: device handler registered [ 3.198589] libphy: Fixed MDIO Bus: probed [ 3.207754] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 3.211156] ehci-pci: EHCI PCI platform driver [ 3.212900] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 3.214969] ohci-pci: OHCI PCI platform driver [ 3.216521] uhci_hcd: USB Universal Host Controller Interface driver [ 3.218938] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.223008] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.224787] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.227516] mousedev: PS/2 mouse device common for all mice [ 3.230702] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.233425] rtc_cmos 00:05: RTC can wake from S4 [ 3.237991] rtc_cmos 00:05: rtc core: registered rtc_cmos as rtc0 [ 3.240537] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.249869] hidraw: raw HID events driver (C) Jiri Kosina [ 3.252095] usbcore: registered new interface driver usbhid [ 3.254141] usbhid: USB HID core driver [ 3.255666] drop_monitor: Initializing network drop monitor service [ 3.258186] Netfilter messages via NETLINK v0.30. [ 3.259999] TCP: cubic registered [ 3.261249] Initializing XFRM netlink socket [ 3.263184] NET: Registered protocol family 10 [ 3.266513] NET: Registered protocol family 17 [ 3.268137] Key type dns_resolver registered [ 3.270220] mce: Using 10 MCE banks [ 3.272111] Loading compiled-in X.509 certificates [ 3.275923] Loaded X.509 cert 'Magrathea: Glacier signing key: e34d0e1b7fcf5b414cce75d36d8482945c781ed6' [ 3.278911] registered taskstats version 1 [ 3.285338] modprobe (72) used greatest stack depth: 13456 bytes left [ 3.292212] modprobe (74) used greatest stack depth: 13184 bytes left [ 3.295283] Key type trusted registered [ 3.302252] Key type encrypted registered [ 3.303832] IMA: No TPM chip found, activating TPM-bypass! (rc=-19) [ 3.308099] BERT: Boot Error Record Table support is disabled. Enable it by using bert_enable as kernel parameter. [ 3.313575] rtc_cmos 00:05: setting system clock to 2025-06-02 12:25:49 UTC (1748867149) [ 3.317080] debug: unmapping init [mem 0xffffffff81da0000-0xffffffff82018fff] [ 3.319785] Write protecting the kernel read-only data: 12288k [ 3.322254] debug: unmapping init [mem 0xffff8800017fe000-0xffff8800017fffff] [ 3.324686] debug: unmapping init [mem 0xffff880001b9b000-0xffff880001bfffff] [ 3.334724] random: systemd: uninitialized urandom read (16 bytes read) [ 3.339047] random: systemd: uninitialized urandom read (16 bytes read) [ 3.341947] random: systemd: uninitialized urandom read (16 bytes read) [ 3.346414] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 3.353098] systemd[1]: Detected virtualization kvm. [ 3.354917] systemd[1]: Detected architecture x86-64. [ 3.356605] systemd[1]: Running in initial RAM disk. [ 3.362432] systemd[1]: No hostname configured. [ 3.363960] systemd[1]: Set hostname to . [ 3.365976] random: systemd: uninitialized urandom read (16 bytes read) [ 3.368478] systemd[1]: Initializing machine ID from random generator. [ 3.433590] dracut-rootfs-g (86) used greatest stack depth: 12784 bytes left [ 3.438963] random: systemd: uninitialized urandom read (16 bytes read) [ 3.441239] random: systemd: uninitialized urandom read (16 bytes read) [ 3.443270] random: systemd: uninitialized urandom read (16 bytes read) [ 3.445173] random: systemd: uninitialized urandom read (16 bytes read) [ 3.449838] random: systemd: uninitialized urandom read (16 bytes read) [ 3.452250] random: systemd: uninitialized urandom read (16 bytes read) [ 3.463548] systemd[1]: Reached target Timers. [ 3.468349] systemd[1]: Reached target Local File Systems. [ 3.472413] tsc: Refined TSC clocksource calibration: 2399.976 MHz [ 3.477067] systemd[1]: Created slice Root Slice. [ 3.481034] systemd[1]: Listening on udev Kernel Socket. [ 3.485557] systemd[1]: Listening on Journal Socket. [ 3.489715] systemd[1]: Created slice System Slice. [ 3.493448] systemd[1]: Reached target Slices. [ 3.499101] systemd[1]: Starting Create list of required static device nodes for the current kernel... [ 3.509778] systemd[1]: Starting Load Kernel Modules... [ 3.515031] systemd[1]: Starting Journal Service... [ 3.519856] systemd[1]: Starting dracut cmdline hook... [ 3.524710] systemd[1]: Starting Setup Virtual Console... [ 3.528532] systemd[1]: Reached target Swap. [ 3.532990] systemd[1]: Listening on udev Control Socket. [ 3.538759] systemd[1]: Reached target Sockets. [ 3.544896] systemd[1]: Started Create list of required static device nodes for the current kernel. [ 3.554717] systemd[1]: Started Load Kernel Modules. [ 3.559122] systemd[1]: Started Setup Virtual Console. [ 3.563418] systemd[1]: Started Journal Service. [ 3.647207] random: fast init done [ 4.080684] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input2 [ 4.180051] libata version 3.00 loaded. [ 4.184395] ata_piix 0000:00:01.1: version 2.13 [ 4.200559] scsi host0: ata_piix [ 4.204226] scsi host1: ata_piix [ 4.207594] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.214470] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 4.406061] ip (343) used greatest stack depth: 12336 bytes left [ 7.661401] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ 8.225329] systemd-journald[103]: Received SIGTERM from PID 1 (systemd). [ 8.517663] SELinux: Disabled at runtime. [ 8.519127] SELinux: Unregistering netfilter hooks [ 8.606582] ip_tables: (C) 2000-2006 Netfilter Core Team [ 8.613634] systemd[1]: Inserted module 'ip_tables' [ 9.138118] systemd-journald[567]: Received request to flush runtime journal from PID 1 [ 9.348209] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 9.399133] cryptd: max_cpu_qlen set to 1000 [ 9.403123] input: PC Speaker as /devices/platform/pcspkr/input/input3 [ 9.427019] rename_device (624) used greatest stack depth: 12272 bytes left [ 9.457528] AVX version of gcm_enc/dec engaged. [ 9.463634] AES CTR mode by8 optimization enabled [ 9.494016] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 9.527233] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ 9.535579] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 9.546645] alg: No test for __generic-gcm-aes-aesni (__driver-generic-gcm-aes-aesni) [ 9.711803] EDAC MC: Ver: 3.0.0 [ 9.725213] EDAC sbridge: Seeking for: PCI ID 8086:0ea0 [ 9.725225] EDAC sbridge: Ver: 1.1.2 [ 12.517964] mount.nfs (769) used greatest stack depth: 10704 bytes left [ 31.139456] device-mapper: uevent: version 1.0.3 [ 31.142327] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com [ 38.518102] libcfs: loading out-of-tree module taints kernel. [ 38.520412] libcfs: module verification failed: signature and/or required key missing - tainting kernel [ 38.580048] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 45.653867] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 46.026208] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 46.031789] alg: No test for adler32 (adler32-zlib) [ 46.999054] Lustre: Lustre: Build Version: 2.16.55_39_g76ed203 [ 47.347710] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 47.353669] LNet: Accept secure, port 988 [ 48.954198] Key type lgssc registered [ 49.570441] Lustre: Echo OBD driver; http://www.lustre.org/ [ 55.325542] icp: module license 'CDDL' taints kernel. [ 55.328332] Disabling lock debugging due to kernel taint [ 57.389744] ZFS: Loaded module v2.1.5-1, ZFS pool version 5000, ZFS filesystem version 5 [ 61.513538] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 70.455958] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 75.166464] LDISKFS-fs (vde): file extents enabled, maximum tree depth=5 [ 75.194194] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 79.356801] LDISKFS-fs (vdf): file extents enabled, maximum tree depth=5 [ 79.367486] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 86.547393] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 92.132728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 92.166225] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 92.175220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 93.300968] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 93.328071] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 93.373977] Lustre: lustre-MDT0000: new disk, initializing [ 93.427562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 93.438858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 93.486479] mount.lustre (7220) used greatest stack depth: 10192 bytes left [ 94.912613] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 102.384031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 102.427505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 102.480561] Lustre: 7238:0:(mgs_llog.c:1347:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 102.504343] Lustre: 8379:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 102.525558] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 102.528979] Lustre: Skipped 1 previous similar message [ 102.574254] Lustre: lustre-MDT0001: new disk, initializing [ 102.610531] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 102.685978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 102.692912] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 104.331266] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 107.074502] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 111.367846] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 111.376132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 111.412640] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 111.420989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 111.524101] Lustre: 9582:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 111.538496] Lustre: lustre-OST0000: new disk, initializing [ 111.543435] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 111.643546] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 113.574103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 113.588239] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 113.601502] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 114.453246] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 122.215112] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 122.222709] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 122.261242] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 122.269115] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 122.331506] Lustre: 10923:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 122.348377] Lustre: lustre-OST0001: new disk, initializing [ 122.352132] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 122.385445] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 122.443070] random: crng init done [ 124.991885] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 131.477483] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 131.481139] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 131.527697] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 131.742885] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 135.002752] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 142.687890] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing check_logdir /tmp/testlogs/ [ 144.742523] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing yml_node [ 147.522009] Lustre: DEBUG MARKER: Client: 2.16.55.39 [ 148.913873] Lustre: DEBUG MARKER: MDS: 2.16.55.39 [ 150.395669] Lustre: DEBUG MARKER: OSS: 2.16.55.39 [ 151.405020] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Jun 2 08:28:17 EDT 2025 [ 160.720344] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 161.379245] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 176.561003] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 176.561650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 176.572472] Lustre: Skipped 1 previous similar message [ 179.019175] LustreError: 14325:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 179.079907] Lustre: server umount lustre-MDT0000 complete [ 180.867464] LustreError: 7230:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748867327 with bad export cookie 4875024472653519527 [ 180.870493] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 180.876769] LustreError: 7230:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 181.569836] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 181.570538] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 181.570539] Lustre: Skipped 3 previous similar messages [ 181.586235] Lustre: Skipped 3 previous similar messages [ 186.577539] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 186.577546] LustreError: 10427:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 186.577548] LustreError: 10427:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 186.589050] Lustre: Skipped 2 previous similar messages [ 186.945168] LustreError: 14595:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 186.947136] LustreError: 14595:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 187.044950] Lustre: server umount lustre-MDT0001 complete [ 188.810361] LustreError: 14866:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 188.815308] LustreError: 14866:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 188.833585] Lustre: server umount lustre-OST0000 complete [ 190.596083] Lustre: server umount lustre-OST0001 complete [ 192.687227] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 195.924128] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 199.747006] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 202.751175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 205.965355] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 205.970857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 208.794148] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 208.802989] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 212.098792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 212.142951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 212.273925] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 212.299937] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 212.347546] Lustre: lustre-MDT0000: new disk, initializing [ 212.414899] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 212.426151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 214.015902] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 219.217795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 219.254521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 219.308952] Lustre: 17797:0:(mgs_llog.c:1347:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 219.328171] Lustre: 18736:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 219.485840] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 219.528755] mount.lustre (18736) used greatest stack depth: 9504 bytes left [ 220.870855] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 224.484535] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 226.072177] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 229.504532] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 229.513958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 229.560474] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 229.573154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 229.686025] Lustre: 19930:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 229.706497] Lustre: lustre-OST0000: new disk, initializing [ 229.710111] Lustre: Skipped 1 previous similar message [ 229.714612] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 229.718257] Lustre: Skipped 2 previous similar messages [ 229.759834] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 229.762521] Lustre: Skipped 1 previous similar message [ 230.928648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 230.934815] Lustre: Skipped 1 previous similar message [ 230.942436] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 231.004221] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 232.108591] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 235.876951] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 237.345510] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 237.439085] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 238.697389] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 238.805828] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 240.997035] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 240.998086] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 240.998380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 241.009200] Lustre: Skipped 1 previous similar message [ 245.662393] LustreError: 21334:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 245.666640] LustreError: 21334:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 245.747753] Lustre: server umount lustre-OST0000 complete [ 249.489684] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 249.495462] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 249.503572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 249.507317] Lustre: Skipped 1 previous similar message [ 253.384234] LustreError: 21605:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 253.386970] LustreError: 21605:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 253.451904] Lustre: server umount lustre-MDT0000 complete [ 255.234007] LustreError: 17789:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748867401 with bad export cookie 4875024472653521060 [ 255.237464] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 255.248227] LustreError: 17789:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 255.480849] Lustre: server umount lustre-MDT0001 complete [ 257.687854] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 08:30:03 (1748867403) [ 258.920573] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 259.762662] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 08:30:05 (1748867405) [ 263.197881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 263.466664] LustreError: 22755:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 263.477345] LustreError: 22755:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 263.506185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 264.917477] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 268.517783] LustreError: 22755:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 269.260327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 269.340510] Lustre: 23616:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 270.948872] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 273.547510] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 275.100951] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 277.877427] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 277.883445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 278.048130] mount.lustre (24726) used greatest stack depth: 9264 bytes left [ 280.129125] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 282.150929] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 298.049643] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 298.051540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 298.051541] Lustre: Skipped 2 previous similar messages [ 298.061107] Lustre: Skipped 2 previous similar messages [ 301.144972] LustreError: 25528:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 301.151246] LustreError: 25528:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 301.222877] Lustre: server umount lustre-OST0000 complete [ 308.950169] Lustre: server umount lustre-MDT0000 complete [ 310.440515] LustreError: 25801:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748867456 with bad export cookie 4875024472653521942 [ 310.444893] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 310.450273] LustreError: 25801:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 310.604799] Lustre: server umount lustre-MDT0001 complete [ 315.074832] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 316.053802] Key type lgssc unregistered [ 316.176649] LNet: 26598:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 317.183770] LNet: Removed LNI 192.168.202.156@tcp [ 325.499042] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 325.682090] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 325.688820] alg: No test for adler32 (adler32-zlib) [ 326.552662] Lustre: Lustre: Build Version: 2.16.55_39_g76ed203 [ 326.624645] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 326.628412] LNet: Accept secure, port 988 [ 328.183431] Key type lgssc registered [ 328.536180] Lustre: Echo OBD driver; http://www.lustre.org/ [ 331.692632] loop: module loaded [ 331.914338] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 334.522549] LDISKFS-fs (loop0): file extents enabled, maximum tree depth=5 [ 334.543310] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 338.973942] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 339.007979] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 339.017880] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 340.154988] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 340.179090] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 340.234029] Lustre: lustre-MDT0000: new disk, initializing [ 340.280629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 340.295227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 341.661084] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 348.252226] LDISKFS-fs (dm-5): file extents enabled, maximum tree depth=5 [ 348.257620] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 348.278747] LDISKFS-fs (dm-5): file extents enabled, maximum tree depth=5 [ 348.283306] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 348.394239] Lustre: 30299:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 348.416227] Lustre: lustre-OST07c6: new disk, initializing [ 348.419792] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 348.422073] Lustre: Skipped 1 previous similar message [ 348.459665] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 350.561591] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 356.543919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 356.550115] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 356.560694] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 441.665071] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 441.665162] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 441.665511] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 444.352834] LustreError: 31849:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 444.407150] Lustre: server umount lustre-OST07c6 complete [ 447.911053] LustreError: 32515:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 447.913263] LustreError: 32515:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 448.033247] Lustre: server umount lustre-MDT0000 complete [ 454.080169] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 454.925495] Key type lgssc unregistered [ 455.030468] LNet: 1008:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 456.037904] LNet: Removed LNI 192.168.202.156@tcp [ 463.514938] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 463.692563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 463.736353] alg: No test for adler32 (adler32-zlib) [ 464.588666] Lustre: Lustre: Build Version: 2.16.55_39_g76ed203 [ 464.654590] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 464.656534] LNet: Accept secure, port 988 [ 466.193483] Key type lgssc registered [ 466.524116] Lustre: Echo OBD driver; http://www.lustre.org/ [ 469.344787] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 469.352372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 470.597064] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 471.910533] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 475.603691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 475.680798] Lustre: 3187:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 475.872211] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 477.407810] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 479.724858] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 481.018365] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 484.189925] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 484.198511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 484.297404] Lustre: 4301:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 484.385626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 486.762633] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 489.136183] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 495.392450] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 502.258905] LustreError: 5050:0:(qsd_reint.c:617:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:2 until pending updates are flushed. [ 506.420233] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 506.427592] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 506.438365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 507.429734] LustreError: lustre-OST0000-osc-MDT0000: operation ost_setattr to node 0@lo failed: rc = -107 [ 507.430140] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 507.431082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 507.446669] LustreError: Skipped 1 previous similar message [ 507.545475] LustreError: 5383:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 507.613192] Lustre: server umount lustre-OST0000 complete [ 510.418268] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 510.422851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 510.427516] Lustre: Skipped 1 previous similar message [ 515.188729] LustreError: 5651:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 515.192915] LustreError: 5651:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 515.217246] Lustre: server umount lustre-MDT0000 complete [ 515.425718] LustreError: 2327:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 515.431178] LustreError: 2327:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 516.625273] LustreError: 5652:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748867662 with bad export cookie 11288438069377784564 [ 516.625723] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 516.634586] LustreError: 5652:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message [ 516.695082] LustreError: 5922:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 516.698796] LustreError: 5922:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 516.805727] Lustre: server umount lustre-MDT0001 complete [ 521.557249] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 522.492756] Key type lgssc unregistered [ 522.592976] LNet: 6448:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 523.602477] LNet: Removed LNI 192.168.202.156@tcp [ 531.213240] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 531.455230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 531.462320] alg: No test for adler32 (adler32-zlib) [ 532.338350] Lustre: Lustre: Build Version: 2.16.55_39_g76ed203 [ 532.427672] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 532.431029] LNet: Accept secure, port 988 [ 533.998906] Key type lgssc registered [ 534.396975] Lustre: Echo OBD driver; http://www.lustre.org/ [ 537.438860] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 537.448971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 538.710867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 540.063435] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 543.837003] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 543.906431] Lustre: 8632:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 544.065114] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 545.379359] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 547.416212] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 548.819881] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 551.754096] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 551.761366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 551.868208] Lustre: 9745:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 551.956451] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 554.087655] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 556.199963] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 562.958202] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 572.977515] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 572.978473] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 572.978781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 576.081921] LustreError: 10550:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 576.149078] Lustre: server umount lustre-OST0000 complete [ 577.985660] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 577.995312] Lustre: Skipped 1 previous similar message [ 578.001975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 578.006833] Lustre: Skipped 2 previous similar messages [ 582.992917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 582.997237] Lustre: Skipped 1 previous similar message [ 583.581050] LustreError: 10823:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 583.583365] LustreError: 10823:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 583.614882] Lustre: server umount lustre-MDT0000 complete [ 585.321244] LustreError: 7751:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748867731 with bad export cookie 2747280562402780061 [ 585.325251] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 585.338349] LustreError: 7751:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 585.420868] LustreError: 11094:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 585.425308] LustreError: 11094:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 585.521437] Lustre: server umount lustre-MDT0001 complete [ 590.099895] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 591.225236] Key type lgssc unregistered [ 591.352817] LNet: 11620:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 592.368871] LNet: Removed LNI 192.168.202.156@tcp [ 601.144520] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 601.371343] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 601.382767] alg: No test for adler32 (adler32-zlib) [ 602.213364] Lustre: Lustre: Build Version: 2.16.55_39_g76ed203 [ 602.272843] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 602.275605] LNet: Accept secure, port 988 [ 603.814007] Key type lgssc registered [ 604.103774] Lustre: Echo OBD driver; http://www.lustre.org/ [ 606.862920] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 606.873645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 608.129176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 609.285927] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 612.731397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 612.807301] Lustre: 13995:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 612.929339] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 614.186860] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 616.257765] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 617.456631] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 620.173466] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 620.183336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 620.270358] Lustre: 15108:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 620.333890] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 622.103299] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 623.947222] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 633.336712] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 637.937150] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 637.942049] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 637.947754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 640.210953] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.56@tcp (stopping) [ 640.213479] Lustre: Skipped 2 previous similar messages [ 643.321781] LustreError: 15910:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 643.345249] LustreError: 13133:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 643.355119] LustreError: 13133:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 643.381583] Lustre: server umount lustre-MDT0000 complete [ 644.765587] LustreError: 15912:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748867790 with bad export cookie 17838951552817634347 [ 644.766731] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 644.775287] LustreError: 15912:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 645.202615] LustreError: 13128:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 645.212612] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.56@tcp (stopping) [ 648.353690] LustreError: 13128:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 648.355198] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 648.355199] Lustre: Skipped 2 previous similar messages [ 648.355945] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 650.357542] LustreError: 15130:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 650.364830] LustreError: 15130:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 650.823988] LustreError: 16184:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 650.826815] LustreError: 16184:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 650.931658] Lustre: server umount lustre-MDT0001 complete [ 668.502189] LustreError: 16462:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 668.505369] LustreError: 16462:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 668.571594] Lustre: server umount lustre-OST0000 complete [ 674.670240] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 675.556577] Key type lgssc unregistered [ 675.671907] LNet: 17394:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 676.681434] LNet: Removed LNI 192.168.202.156@tcp [ 685.767742] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 685.952171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 685.991366] alg: No test for adler32 (adler32-zlib) [ 686.851055] Lustre: Lustre: Build Version: 2.16.55_39_g76ed203 [ 686.931844] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 686.935281] LNet: Accept secure, port 988 [ 688.478426] Key type lgssc registered [ 688.849188] Lustre: Echo OBD driver; http://www.lustre.org/ [ 691.725163] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 691.735527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 693.002395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 694.293429] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 697.617416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 697.671197] Lustre: 19768:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 697.766042] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 698.791801] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 700.448376] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 701.535913] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 703.857519] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 703.864843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 703.943061] Lustre: 20879:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 704.006174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 705.600024] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 707.346351] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 718.008146] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 722.032961] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 722.035898] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 722.040894] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 723.008939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 723.009584] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 723.011617] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 723.020131] Lustre: Skipped 1 previous similar message [ 726.388387] LustreError: 21680:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 726.440361] Lustre: server umount lustre-OST0000 complete [ 742.817163] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 742.820599] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 742.825969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 742.829595] Lustre: Skipped 1 previous similar message [ 744.650110] LustreError: 22161:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 744.653282] LustreError: 22161:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 744.685696] Lustre: server umount lustre-MDT0000 complete [ 745.847170] LustreError: 22163:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748867892 with bad export cookie 14562032299222627230 [ 745.848296] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 745.856551] LustreError: 22163:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 751.902420] LustreError: 22434:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 751.904683] LustreError: 22434:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 751.979888] Lustre: server umount lustre-MDT0001 complete [ 755.399168] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 756.053310] Key type lgssc unregistered [ 756.161975] LNet: 22963:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 757.174027] LNet: Removed LNI 192.168.202.156@tcp [ 763.929253] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 764.103392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 764.108858] alg: No test for adler32 (adler32-zlib) [ 764.926368] Lustre: Lustre: Build Version: 2.16.55_39_g76ed203 [ 764.978194] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 764.980383] LNet: Accept secure, port 988 [ 766.510516] Key type lgssc registered [ 766.741415] Lustre: Echo OBD driver; http://www.lustre.org/ [ 768.848404] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 768.855795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 770.011912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 770.863382] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 773.370574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 773.414576] Lustre: 25333:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 773.504073] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 774.380701] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 775.855354] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 776.941668] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 779.046397] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 779.051117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 779.126101] Lustre: 26447:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 779.178408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 780.180361] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 780.632508] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 782.164783] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 784.993812] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 785.567158] LNetError: Refusing connection from 192.168.202.156 for 127.0.0.2@tcp: No matching NI [ 785.569742] LNetError: 23703:0:(socklnd_cb.c:1794:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 785.572968] LNetError: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 785.949170] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 796.418113] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2025-06-02 8h39m02s [ 796.857608] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 798.210869] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 798.211754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 798.213168] LNetError: Refusing connection from 192.168.202.156 for 127.0.0.2@tcp: No matching NI [ 798.213221] LNetError: 23706:0:(socklnd_cb.c:1794:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 798.213238] LNetError: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 798.227359] Lustre: Skipped 1 previous similar message [ 799.234581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 799.237300] Lustre: Skipped 6 previous similar messages [ 803.554871] LustreError: 27643:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 803.603381] Lustre: server umount lustre-MDT0000 complete [ 804.240879] LNetError: Refusing connection from 192.168.202.156 for 127.0.0.2@tcp: No matching NI [ 804.243772] LNetError: 23704:0:(socklnd_cb.c:1794:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 804.247159] LNetError: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 804.716014] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 805.244415] LustreError: 26337:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 805.252493] LustreError: 26337:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 807.001703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 807.059492] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 807.149850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 807.171580] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 808.149448] LNetError: Refusing connection from 192.168.202.156 for 127.0.0.2@tcp: No matching NI [ 808.152623] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 808.159516] LNetError: 23705:0:(socklnd_cb.c:1794:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 808.166236] LNetError: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 808.169048] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 809.206018] Lustre: DEBUG MARKER: Wait for df (3910) ... [ 810.828910] Lustre: DEBUG MARKER: done [ 812.145268] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 812.148581] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 812.156840] Lustre: Skipped 1 previous similar message [ 812.161936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 812.166058] Lustre: Skipped 1 previous similar message [ 813.154827] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 813.842367] LustreError: 28970:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 813.847606] LustreError: 28970:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 813.914358] Lustre: server umount lustre-OST0000 complete [ 817.217499] LustreError: 29240:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 817.220148] LustreError: 29240:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 817.247869] Lustre: server umount lustre-MDT0000 complete [ 818.161346] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 818.161628] LustreError: 24495:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 818.161631] LustreError: 24495:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 818.178747] Lustre: Skipped 1 previous similar message [ 818.180951] LNetError: Refusing connection from 192.168.202.156 for 127.0.0.2@tcp: No matching NI [ 818.186465] LNetError: 23703:0:(socklnd_cb.c:1794:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 818.190197] LNetError: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 818.448504] LustreError: 24455:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748867964 with bad export cookie 9709114347161267094 [ 818.449709] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 818.456866] LustreError: 24455:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 820.503698] LustreError: 29507:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 820.505981] LustreError: 29507:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 820.573188] Lustre: server umount lustre-MDT0001 complete [ 824.309057] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 825.186341] Key type lgssc unregistered [ 825.311949] LNet: 30032:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 826.319943] LNet: Removed LNI 192.168.202.156@tcp [ 838.115120] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 842.872144] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 843.001251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 843.011426] alg: No test for adler32 (adler32-zlib) [ 843.838764] Lustre: Lustre: Build Version: 2.16.55_39_g76ed203 [ 843.895845] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 843.898182] LNet: Accept secure, port 988 [ 845.433603] Key type lgssc registered [ 845.734909] Lustre: Echo OBD driver; http://www.lustre.org/ [ 848.332639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 850.403205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 852.604366] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 852.610861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 854.812077] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 854.820352] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 857.103744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 857.126528] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 857.132322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 858.218816] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 858.235372] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 858.270605] Lustre: lustre-MDT0000: new disk, initializing [ 858.301039] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 858.309659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 859.402856] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 863.984666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 864.012498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 864.060647] Lustre: 2410:0:(mgs_llog.c:1347:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 864.075633] Lustre: 3347:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 864.094077] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 864.096380] Lustre: Skipped 1 previous similar message [ 864.143529] Lustre: lustre-MDT0001: new disk, initializing [ 864.181936] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 864.200415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 864.208073] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 865.367165] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 868.498170] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 869.907848] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 872.600253] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 872.610146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 872.636240] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 872.642931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 872.744301] Lustre: 4553:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 872.759778] Lustre: lustre-OST0000: new disk, initializing [ 872.762680] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 872.791006] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 874.437165] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 874.531158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 874.537934] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 874.564498] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 877.501406] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 878.688560] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 878.757254] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 879.907113] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 879.976576] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 884.562042] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 884.564483] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 884.564873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 884.583827] LustreError: Skipped 1 previous similar message [ 886.672726] LustreError: 5953:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 886.738686] Lustre: server umount lustre-OST0000 complete [ 889.201833] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 889.207619] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 889.212555] Lustre: Skipped 1 previous similar message [ 889.217043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 889.219805] Lustre: Skipped 1 previous similar message [ 894.198449] LustreError: 6223:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 894.200888] LustreError: 6223:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 894.246800] Lustre: server umount lustre-MDT0000 complete [ 895.487252] LustreError: 4585:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748868041 with bad export cookie 18206759426668096065 [ 895.490112] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 895.501251] LustreError: 4585:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 895.549161] LustreError: 6494:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 895.552034] LustreError: 6494:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 895.642916] Lustre: server umount lustre-MDT0001 complete [ 898.523021] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 08:40:44 (1748868044) [ 900.709565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 900.910567] LustreError: 7375:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 900.920268] LustreError: 7375:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 900.942645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 901.849183] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 904.518211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 904.567867] Lustre: 8241:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 905.523442] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 907.050681] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 908.057365] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 910.437759] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 910.444893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 910.536695] Lustre: 9353:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 910.596161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 910.599221] Lustre: Skipped 1 previous similar message [ 912.237530] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 913.752535] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 916.964387] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2025-06-02 8h41m02s [ 917.509350] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 919.893741] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 920.696018] Lustre: DEBUG MARKER: Creating a test file and stat it [ 920.708018] LustreError: 7370:0:(mdt_handler.c:2326:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 924.949827] LNetError: Refusing connection from 192.168.202.156 for 127.0.0.2@tcp: No matching NI [ 924.953720] LNetError: 32629:0:(socklnd_cb.c:1794:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 924.957144] LNetError: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 940.725850] Lustre: lustre-MDT0000: Client 69883ecf-9976-4bba-a26c-2659aad888d1 (at 192.168.202.56@tcp) reconnecting [ 960.737533] LustreError: 7370:0:(mdt_handler.c:2326:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 960.767986] Lustre: lustre-MDT0000: Client 69883ecf-9976-4bba-a26c-2659aad888d1 (at 192.168.202.56@tcp) reconnecting [ 961.273949] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 961.975439] Lustre: DEBUG MARKER: done [ 964.673185] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 964.673461] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 964.673939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 964.673940] Lustre: Skipped 1 previous similar message [ 964.685133] Lustre: Skipped 2 previous similar messages [ 969.274643] LustreError: 11159:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 969.278040] LustreError: 11159:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 969.336038] Lustre: server umount lustre-OST0000 complete [ 974.689070] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 974.689468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 974.689478] Lustre: Skipped 1 previous similar message [ 974.697054] Lustre: Skipped 1 previous similar message [ 974.698916] LNetError: Refusing connection from 192.168.202.156 for 127.0.0.2@tcp: No matching NI [ 974.701346] LNetError: 32632:0:(socklnd_cb.c:1794:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 974.704548] LNetError: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 976.525204] LustreError: 11432:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 976.527809] LustreError: 11432:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 976.553657] Lustre: server umount lustre-MDT0000 complete [ 977.741511] LustreError: 7357:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748868123 with bad export cookie 18206759426668096933 [ 977.742750] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 977.749958] LustreError: 7357:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 977.883118] Lustre: server umount lustre-MDT0001 complete [ 981.359554] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 982.025396] Key type lgssc unregistered [ 982.126931] LNet: 12227:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 983.133068] LNet: Removed LNI 192.168.202.156@tcp [ 993.114857] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 997.977481] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 998.160058] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 998.165874] alg: No test for adler32 (adler32-zlib) [ 999.000428] Lustre: Lustre: Build Version: 2.16.55_39_g76ed203 [ 999.061629] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 999.063826] LNet: Accept secure, port 988 [ 1000.599431] Key type lgssc registered [ 1000.858425] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1003.297818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1005.216300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1007.018701] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1007.023549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1009.088451] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1009.093579] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1011.137571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1011.157100] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1011.163016] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1012.240986] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1012.257120] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1012.293090] Lustre: lustre-MDT0000: new disk, initializing [ 1012.325137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1012.332497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1013.205764] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1017.325335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1017.361086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1017.410053] Lustre: 17031:0:(mgs_llog.c:1347:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1017.424597] Lustre: 17968:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1017.446801] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1017.450309] Lustre: Skipped 1 previous similar message [ 1017.483901] Lustre: lustre-MDT0001: new disk, initializing [ 1017.521479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1017.537328] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1017.542989] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1018.487996] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1021.464816] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1022.604465] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1024.805078] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1024.810081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1024.829584] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1024.834470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1024.904932] Lustre: 19165:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1024.917421] Lustre: lustre-OST0000: new disk, initializing [ 1024.919855] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1024.946112] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1026.347135] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1029.193505] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1030.230894] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1034.576945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1034.582043] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1034.617257] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1035.394712] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec [ 1036.389572] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1036.449060] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1039.601463] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1039.601525] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1039.601847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1039.611310] Lustre: Skipped 1 previous similar message [ 1043.084647] LustreError: 20605:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1043.140267] Lustre: server umount lustre-OST0000 complete [ 1044.609258] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1044.610789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1044.610790] Lustre: Skipped 1 previous similar message [ 1044.622756] Lustre: Skipped 1 previous similar message [ 1049.617047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1049.619193] Lustre: Skipped 2 previous similar messages [ 1050.426638] LustreError: 20875:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1050.429468] LustreError: 20875:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1050.481452] Lustre: server umount lustre-MDT0000 complete [ 1051.761459] LustreError: 17022:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748868197 with bad export cookie 17556929739133284031 [ 1051.762126] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1051.773334] LustreError: 17022:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1051.841935] LustreError: 21146:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1051.844392] LustreError: 21146:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1051.943584] Lustre: server umount lustre-MDT0001 complete [ 1054.478462] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 08:43:20 (1748868200) [ 1054.938155] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1055.455054] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 08:43:21 (1748868201) [ 1058.300560] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 1059.292087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1059.439675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1059.733773] Lustre: Failing over lustre-MDT0000 [ 1059.736487] LustreError: 22406:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1059.741073] LustreError: 22406:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1059.745461] LustreError: 22406:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 0, retries 0, failed: rc = -5 [ 1059.811436] LustreError: 22474:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1059.813888] LustreError: 22474:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1059.871600] Lustre: server umount lustre-MDT0000 complete [ 1061.667605] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 08:43:27 (1748868207) [ 1064.029043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1064.276108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1065.513079] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1068.641777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1068.698269] Lustre: 24078:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1069.776877] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1071.282456] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1072.188001] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1074.149115] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1074.155449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1074.241739] Lustre: 25187:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1074.294997] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1074.298561] Lustre: Skipped 1 previous similar message [ 1075.689270] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1077.086043] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1083.639445] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1088.305142] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1088.306540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1088.311282] Lustre: Skipped 2 previous similar messages [ 1091.246367] LustreError: 26118:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1091.249643] LustreError: 26118:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1091.264756] Lustre: server umount lustre-MDT0000 complete [ 1092.443390] LustreError: 25212:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748868238 with bad export cookie 17556929739133285179 [ 1092.445597] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1092.453349] LustreError: 25212:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1092.572304] Lustre: server umount lustre-MDT0001 complete [ 1093.559465] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1094.856391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1097.384007] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1102.313022] LustreError: 14833:0:(client.c:1383:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8800974bc700 x1833821164067328/t0(0) o250->MGC192.168.202.156@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1102.429362] LustreError: 25203:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1102.438944] LustreError: 25203:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 1102.455143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1102.484339] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1103.375912] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1105.907255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1105.941447] Lustre: 28387:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1106.001548] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1106.988881] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1108.036144] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 1108.503495] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1109.033428] Lustre: 14834:0:(client.c:2451:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1748868239/real 1748868239] req@ffff88009c31f100 x1833821164066944/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1748868255 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1109.424113] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1113.010443] Lustre: 14836:0:(client.c:2451:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1748868243/real 1748868243] req@ffff8800974bc380 x1833821164067072/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1748868259 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1113.041211] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1113.044143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1113.044144] Lustre: Skipped 2 previous similar messages [ 1113.049707] Lustre: Skipped 2 previous similar messages [ 1114.040905] Lustre: 14837:0:(client.c:2451:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1748868244/real 1748868244] req@ffff8800974be300 x1833821164067200/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1748868260 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1117.201628] LustreError: 29240:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1117.204599] LustreError: 29240:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1117.225441] Lustre: server umount lustre-MDT0000 complete [ 1118.039071] Lustre: 14834:0:(client.c:2451:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1748868248/real 1748868248] req@ffff8800974bce00 x1833821164067456/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1748868264 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1118.049781] LustreError: 27520:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1118.058010] LustreError: 27520:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1118.352562] LustreError: 23201:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748868264 with bad export cookie 17556929739133287307 [ 1118.354089] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1118.361299] LustreError: 23201:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1118.488411] Lustre: server umount lustre-MDT0001 complete [ 1120.333233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1122.991927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1128.057851] LustreError: 14833:0:(client.c:1383:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8800974bc380 x1833821164086144/t0(0) o250->MGC192.168.202.156@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1128.163126] LustreError: 25203:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1128.199519] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1129.101569] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1131.416515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1131.448027] Lustre: 31508:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1131.502444] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1132.360974] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1133.667269] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1134.448377] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1136.546923] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 1136.549230] Lustre: Skipped 1 previous similar message [ 1139.545471] Lustre: 14835:0:(client.c:2451:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1748868269/real 1748868269] req@ffff88009f979500 x1833821164086016/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1748868285 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1141.522083] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1141.527338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1141.530034] Lustre: Skipped 2 previous similar messages [ 1146.973445] LustreError: 32362:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1146.975404] LustreError: 32362:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1147.004352] Lustre: server umount lustre-MDT0000 complete [ 1148.006145] LustreError: 23202:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748868294 with bad export cookie 17556929739133288791 [ 1148.007189] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1148.013628] LustreError: 23202:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1148.109655] Lustre: server umount lustre-MDT0001 complete [ 1149.877824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1150.967653] Lustre: DEBUG MARKER: files compared the same [ 1161.592941] Lustre: server umount lustre-OST0000 complete [ 1166.381558] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 1167.000372] Key type lgssc unregistered [ 1167.091745] LNet: 1929:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1168.098596] LNet: Removed LNI 192.168.202.156@tcp [ 1174.353317] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 1174.524315] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1174.529892] alg: No test for adler32 (adler32-zlib) [ 1175.338203] Lustre: Lustre: Build Version: 2.16.55_39_g76ed203 [ 1175.391589] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 1175.393605] LNet: Accept secure, port 988 [ 1176.919416] Key type lgssc registered [ 1177.175781] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1179.332322] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1179.341234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1180.536541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1181.408851] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug +malloc all [ 1183.806700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1183.847657] Lustre: 4297:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1183.918732] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1184.818663] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug +malloc all [ 1186.164698] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1187.032871] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1189.211106] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1189.216224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1189.288848] Lustre: 5419:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1189.324509] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1190.322399] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1190.750039] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug +malloc all [ 1192.216753] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1199.328833] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1199.329000] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1199.336375] Lustre: Skipped 1 previous similar message [ 1199.337901] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1202.327129] LustreError: 6217:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1202.375456] Lustre: server umount lustre-OST0000 complete [ 1203.936741] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1203.938929] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1203.943221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1203.945445] Lustre: Skipped 1 previous similar message [ 1209.344937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1209.347755] Lustre: Skipped 1 previous similar message [ 1209.485474] LustreError: 6486:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1209.488263] LustreError: 6486:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1209.503227] Lustre: server umount lustre-MDT0000 complete [ 1210.481059] LustreError: 3421:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748868356 with bad export cookie 6723688295484615530 [ 1210.481955] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1210.488671] LustreError: 3421:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1210.532511] LustreError: 6757:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1210.534964] LustreError: 6757:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1210.590554] Lustre: server umount lustre-MDT0001 complete [ 1213.672687] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 1214.249454] Key type lgssc unregistered [ 1214.338795] LNet: 7283:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1215.344624] LNet: Removed LNI 192.168.202.156@tcp [ 1222.124874] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 1222.251584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1222.256520] alg: No test for adler32 (adler32-zlib) [ 1223.069197] Lustre: Lustre: Build Version: 2.16.55_39_g76ed203 [ 1223.122174] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 1223.124077] LNet: Accept secure, port 988 [ 1224.658415] Key type lgssc registered [ 1224.884110] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1226.980426] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1226.984870] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1226.989483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1244.056557] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1250.058754] LustreError: 8772:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.156@tcp: failed processing log, type 4: rc = -110 [ 1280.075494] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1281.316720] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1282.590613] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1284.722546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1284.755096] Lustre: *** cfs_fail_loc=706, val=0*** [ 1284.872736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1284.890544] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1285.689879] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1288.075530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1288.117939] Lustre: 10886:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1288.198947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1289.100644] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1290.562929] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1291.457366] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1293.184831] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1293.187730] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1293.192667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1294.065494] LustreError: 11670:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 1294.116003] Lustre: server umount lustre-OST0000 complete [ 1298.193223] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1298.196641] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1298.202005] Lustre: Skipped 1 previous similar message [ 1298.204120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1298.208849] Lustre: Skipped 1 previous similar message [ 1301.257554] LustreError: 11937:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1301.259785] LustreError: 11937:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1301.276122] Lustre: server umount lustre-MDT0000 complete [ 1302.288341] LustreError: 11940:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748868448 with bad export cookie 4815182106550217188 [ 1302.289742] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1302.298116] LustreError: 11940:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1302.343669] LustreError: 12210:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1302.345338] LustreError: 12210:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1302.417513] Lustre: server umount lustre-MDT0001 complete [ 1304.421107] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 1304.996046] Key type lgssc unregistered [ 1305.090939] LNet: 12734:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1306.096613] LNet: Removed LNI 192.168.202.156@tcp [ 1312.995123] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 1313.117952] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1313.150768] alg: No test for adler32 (adler32-zlib) [ 1313.972903] Lustre: Lustre: Build Version: 2.16.55_39_g76ed203 [ 1314.023574] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 1314.025492] LNet: Accept secure, port 988 [ 1315.550424] Key type lgssc registered [ 1315.770444] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1317.765413] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1317.771947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1319.645148] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1321.802140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1321.898988] Lustre: 15069:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1321.983772] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1322.838220] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1325.136939] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1325.141522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1325.196811] Lustre: 15937:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1325.236276] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1326.532122] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1328.817758] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1328.823607] Lustre: MGS: Client 8a74b35b-091e-4710-a5e2-fbb9c790c95c (at 0@lo) reconnecting [ 1328.895868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1328.916144] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1329.723058] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1338.897516] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1338.900221] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1338.904724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1341.906398] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1341.906856] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1341.907176] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1343.046405] LustreError: 17461:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1343.116349] Lustre: server umount lustre-OST0000 complete [ 1346.913028] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1346.913099] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1346.913585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1346.913586] Lustre: Skipped 1 previous similar message [ 1350.344341] LustreError: 17732:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1350.347144] LustreError: 17732:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1350.354814] Lustre: server umount lustre-MDT0000 complete [ 1351.471870] LustreError: 18005:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1351.473760] LustreError: 18005:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1351.531828] Lustre: server umount lustre-MDT0001 complete [ 1352.629508] Lustre: server umount MGS complete [ 1355.618768] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 08:48:21 (1748868501) [ 1363.928587] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 1366.104177] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 1368.927327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1370.856062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1372.513756] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1372.517410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1374.215579] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1374.219902] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1376.042005] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1376.054692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1376.873828] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1380.333940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1380.362337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1380.462489] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1380.470011] Lustre: 23894:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1380.483599] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1380.517792] Lustre: lustre-MDT0001: new disk, initializing [ 1380.539851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1380.548535] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1380.552268] Lustre: 23932:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1380.556136] LustreError: 23932:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1381.355390] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1385.002455] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1385.006079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1385.025435] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1385.028458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1385.095500] Lustre: 24835:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1385.106662] Lustre: lustre-OST0000: new disk, initializing [ 1385.109379] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1385.126265] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1386.490326] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1389.339047] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1391.339841] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1391.345795] Lustre: MGS: Client 3ef44bb9-4630-4b4a-9faf-b05b15851d5b (at 0@lo) reconnecting [ 1391.359002] Lustre: 23060:0:(mgs_llog.c:1347:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1391.367025] LustreError: 23060:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 1391.379050] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1391.414890] Lustre: lustre-MDT0000: new disk, initializing [ 1391.419801] Lustre: 25804:0:(client.c:1609:after_reply()) @@@ resending request on EINPROGRESS req@ffff880094e7e300 x1833821494347648/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1748868553 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1391.462613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1392.354173] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1392.469167] Lustre: 26057:0:(client.c:1609:after_reply()) @@@ resending request on EINPROGRESS req@ffff880094ee7100 x1833821494351104/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1748868554 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1393.431709] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1400.440902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1400.445054] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1401.479393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 1401.483298] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 1401.493368] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 1420.497543] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1420.501737] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1420.508510] Lustre: Skipped 1 previous similar message [ 1420.511197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1420.513657] Lustre: Skipped 1 previous similar message [ 1426.157528] LustreError: 26670:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 1426.159411] LustreError: 26670:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1426.216044] Lustre: server umount lustre-OST0000 complete [ 1428.017878] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1428.018365] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1428.018367] Lustre: Skipped 1 previous similar message [ 1428.018726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1428.018728] Lustre: Skipped 3 previous similar messages [ 1428.031248] LustreError: Skipped 1 previous similar message [ 1433.400555] LustreError: 26941:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 1433.402590] LustreError: 26941:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1433.424869] Lustre: server umount lustre-MDT0000 complete [ 1440.703753] Lustre: server umount lustre-MDT0001 complete [ 1441.805659] LustreError: 27482:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1441.808575] LustreError: 27482:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1441.826676] Lustre: server umount MGS complete [ 1444.728509] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 08:49:50 (1748868590) [ 1449.864853] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 1450.498421] Key type lgssc unregistered [ 1450.590836] LNet: 29069:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1451.596799] LNet: Removed LNI 192.168.202.156@tcp [ 1456.639140] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 1456.819628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1456.824839] alg: No test for adler32 (adler32-zlib) [ 1457.634045] Lustre: Lustre: Build Version: 2.16.55_39_g76ed203 [ 1457.684168] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 1457.688760] LNet: Accept secure, port 988 [ 1459.222535] Key type lgssc registered [ 1459.444696] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1461.322631] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1461.327127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1462.455829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1463.284553] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1465.754976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1465.783952] Lustre: 31180:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1465.886842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1466.687175] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1468.041033] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1468.912099] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1470.631983] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1470.636267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1470.693549] Lustre: 32291:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1470.727409] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1471.948862] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1472.730298] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 1473.258074] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1481.729322] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1481.730042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1481.736271] Lustre: Skipped 1 previous similar message [ 1486.059376] LustreError: 627:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1486.107516] Lustre: server umount lustre-OST0000 complete [ 1490.912936] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1490.915904] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1490.920176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1490.921932] Lustre: Skipped 1 previous similar message [ 1493.194702] LustreError: 928:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1493.197546] LustreError: 928:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1493.221619] Lustre: server umount lustre-MDT0000 complete [ 1494.240208] LustreError: 929:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748868640 with bad export cookie 3272822099749672063 [ 1494.241664] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1494.249116] LustreError: 929:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1494.298021] LustreError: 1201:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1494.299970] LustreError: 1201:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1494.364872] Lustre: server umount lustre-MDT0001 complete [ 1497.270180] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 1497.876063] Key type lgssc unregistered [ 1497.964988] LNet: 1725:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1498.971201] LNet: Removed LNI 192.168.202.156@tcp [ 1503.677825] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 1503.786354] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1503.817567] alg: No test for adler32 (adler32-zlib) [ 1504.631509] Lustre: Lustre: Build Version: 2.16.55_39_g76ed203 [ 1504.678444] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 1504.679677] LNet: Accept secure, port 988 [ 1506.206417] Key type lgssc registered [ 1506.391076] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1507.735881] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1507.736070] LustreError: 2798:0:(tgt_mount.c:2221:server_fill_super()) cfs_race id 716 sleeping [ 1507.740460] Lustre: Skipped 1 previous similar message [ 1507.741660] LustreError: 2797:0:(tgt_mount.c:2221:server_fill_super()) cfs_fail_race id 716 waking [ 1507.743878] LustreError: 2798:0:(tgt_mount.c:2221:server_fill_super()) cfs_fail_race id 716 awake: rc=4992 [ 1507.747066] LustreError: 2798:0:(tgt_mount.c:2233:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1507.749891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1507.761231] LustreError: 2798:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1508.885930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1510.805576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1510.831360] Lustre: 3394:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1510.892849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1511.658846] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1512.682513] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing lsmod [ 1513.452671] LustreError: 4201:0:(tgt_mount.c:2221:server_fill_super()) cfs_race id 716 sleeping [ 1513.452674] LustreError: 4202:0:(tgt_mount.c:2221:server_fill_super()) cfs_fail_race id 716 waking [ 1513.455309] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1513.456893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1513.467096] LustreError: 4201:0:(tgt_mount.c:2221:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 [ 1513.470330] LustreError: 4201:0:(tgt_mount.c:2233:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1513.473324] LustreError: 4201:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1513.513885] Lustre: 4202:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1513.556383] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1514.285405] LustreError: 2870:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1514.286783] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1514.287303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1514.294329] LustreError: 2870:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1514.297427] LustreError: 2870:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 5, retries 0, failed: rc = -5 [ 1519.297031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1519.299663] Lustre: Skipped 4 previous similar messages [ 1520.347777] LustreError: 4440:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1520.363302] Lustre: server umount lustre-MDT0000 complete [ 1521.298176] LustreError: 3413:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1521.307889] LustreError: 3413:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 1521.367205] LustreError: 4442:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748868667 with bad export cookie 8226633897007622900 [ 1521.368692] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1521.376329] LustreError: 4442:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1523.312463] LustreError: 3401:0:(client.c:1373:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88009bf34e00 x1833821693557376/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1523.322617] LustreError: 4727:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1523.326014] LustreError: 4727:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1523.386172] Lustre: server umount lustre-MDT0001 complete [ 1534.319383] LustreError: 4993:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1534.321654] LustreError: 4993:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1534.381731] Lustre: server umount lustre-OST0000 complete [ 1536.889326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1537.016162] LustreError: 5621:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1537.034187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1537.872341] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1540.129776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1540.156871] Lustre: 6481:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1541.029026] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1542.343722] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1543.185746] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1545.010985] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1545.014873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1545.070696] Lustre: 7592:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1545.102532] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1545.105083] Lustre: Skipped 1 previous similar message [ 1546.312275] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1547.104179] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 1547.527944] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1555.105454] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1555.106045] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1555.112315] Lustre: Skipped 3 previous similar messages [ 1555.114439] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1556.389117] LustreError: 8390:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1556.390994] LustreError: 8390:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1556.446536] Lustre: server umount lustre-OST0000 complete [ 1559.653332] Lustre: server umount lustre-MDT0000 complete [ 1560.113087] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1560.113089] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1560.114765] LustreError: 5621:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1560.672549] LustreError: 5602:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748868706 with bad export cookie 8226633897007623551 [ 1560.673949] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1560.680057] LustreError: 5602:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1560.736012] LustreError: 8924:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1560.738880] LustreError: 8924:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 1560.836941] Lustre: server umount lustre-MDT0001 complete [ 1563.942729] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 1564.522526] Key type lgssc unregistered [ 1564.619644] LNet: 9452:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1565.624667] LNet: Removed LNI 192.168.202.156@tcp [ 1572.659506] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 1572.829525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1572.833359] alg: No test for adler32 (adler32-zlib) [ 1573.655561] Lustre: Lustre: Build Version: 2.16.55_39_g76ed203 [ 1573.707974] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 1573.709970] LNet: Accept secure, port 988 [ 1575.239607] Key type lgssc registered [ 1575.489233] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1577.462762] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1577.468884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1578.645578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1579.555846] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1581.917760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1581.944630] Lustre: 11829:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1582.014875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1582.797377] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1584.129924] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1585.021521] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1586.927918] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1586.931737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1586.995155] Lustre: 12940:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1587.041221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1588.043404] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 1588.372128] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1589.742196] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1597.168350] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1602.065046] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1602.065354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1602.070394] Lustre: Skipped 1 previous similar message [ 1602.072247] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1604.363669] LustreError: 13880:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1604.405782] Lustre: server umount lustre-OST0000 complete [ 1607.040947] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1607.045260] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1607.052457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1607.054704] Lustre: Skipped 1 previous similar message [ 1611.342518] LustreError: 14154:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1611.344554] LustreError: 14154:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1611.357325] Lustre: server umount lustre-MDT0000 complete [ 1612.080776] LustreError: 12957:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1612.087608] LustreError: 12957:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 1612.313017] LustreError: 10950:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748868758 with bad export cookie 1638344495555586834 [ 1612.314156] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1612.319663] LustreError: 10950:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1612.371980] LustreError: 14425:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1612.377546] LustreError: 14425:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1612.449444] Lustre: server umount lustre-MDT0001 complete [ 1615.524916] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 1616.136658] Key type lgssc unregistered [ 1616.227887] LNet: 14950:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1617.233593] LNet: Removed LNI 192.168.202.156@tcp [ 1622.186048] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 1622.297820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1622.326477] alg: No test for adler32 (adler32-zlib) [ 1623.133704] Lustre: Lustre: Build Version: 2.16.55_39_g76ed203 [ 1623.179745] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 1623.181701] LNet: Accept secure, port 988 [ 1624.708442] Key type lgssc registered [ 1624.938368] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1626.800881] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1626.806105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1627.943514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1628.797524] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1631.087433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1631.116434] Lustre: 17060:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1631.193001] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1631.967364] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1633.252999] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1634.025209] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1635.708651] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1635.712519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1635.774498] Lustre: 18169:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1635.817576] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1637.053455] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1637.816304] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 1638.328621] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1645.824942] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1645.825439] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1645.826089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1647.277017] LustreError: 18967:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1647.332509] Lustre: server umount lustre-OST0000 complete [ 1650.832951] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1650.833260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1650.833262] Lustre: Skipped 1 previous similar message [ 1650.840326] Lustre: Skipped 2 previous similar messages [ 1654.424071] LustreError: 19235:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1654.426620] LustreError: 19235:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1654.441912] Lustre: server umount lustre-MDT0000 complete [ 1655.410117] LustreError: 16184:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748868801 with bad export cookie 11274540962659777025 [ 1655.411545] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1655.418759] LustreError: 16184:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1655.471604] LustreError: 19504:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1655.474455] LustreError: 19504:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1655.534157] Lustre: server umount lustre-MDT0001 complete [ 1658.466036] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 1659.004072] Key type lgssc unregistered [ 1659.091697] LNet: 20027:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1660.098815] LNet: Removed LNI 192.168.202.156@tcp [ 1667.174800] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 1667.300001] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1667.305147] alg: No test for adler32 (adler32-zlib) [ 1668.129696] Lustre: Lustre: Build Version: 2.16.55_39_g76ed203 [ 1668.184222] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 1668.186605] LNet: Accept secure, port 988 [ 1669.716396] Key type lgssc registered [ 1669.961402] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1671.955896] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1671.961677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1673.094856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1673.843698] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1676.007348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1676.034579] Lustre: 22397:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1676.105562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1676.863350] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1678.182980] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1679.104921] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1680.902066] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1680.907997] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1680.977656] Lustre: 23506:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1681.041094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1682.391115] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1683.041297] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 1683.699787] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1691.496569] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 1692.461949] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 1692.463640] Lustre: Skipped 2 previous similar messages [ 1693.638151] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 1693.640947] Lustre: Skipped 2 previous similar messages [ 1694.752005] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 1694.758008] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1701.078015] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.56@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 1701.082828] Lustre: Skipped 2 previous similar messages [ 1701.980534] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.56@tcp 192.168.0.[2,10]@tcp [ 1709.528799] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.56/24@tcp in log lustre-MDT0000 [ 1709.532080] Lustre: Skipped 2 previous similar messages [ 1711.473351] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.56/24@tcp [ 1711.475159] Lustre: Skipped 1 previous similar message [ 1721.089409] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1721.090321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1721.096187] Lustre: Skipped 1 previous similar message [ 1725.228650] LustreError: 25919:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1725.276878] Lustre: server umount lustre-OST0000 complete [ 1731.105136] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1731.105531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1731.105532] Lustre: Skipped 1 previous similar message [ 1731.113077] Lustre: Skipped 1 previous similar message [ 1732.330628] LustreError: 26189:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1732.333051] LustreError: 26189:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1732.349053] Lustre: server umount lustre-MDT0000 complete [ 1733.369040] LustreError: 21522:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748868879 with bad export cookie 11414080707888360260 [ 1733.370032] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1733.374879] LustreError: 21522:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1733.423079] LustreError: 26459:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1733.425270] LustreError: 26459:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1733.485631] Lustre: server umount lustre-MDT0001 complete [ 1736.691260] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 1737.355406] Key type lgssc unregistered [ 1737.448790] LNet: 26984:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1738.454854] LNet: Removed LNI 192.168.202.156@tcp [ 1746.089538] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 1746.193995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1746.198702] alg: No test for adler32 (adler32-zlib) [ 1747.044049] Lustre: Lustre: Build Version: 2.16.55_39_g76ed203 [ 1747.107893] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 1747.110800] LNet: Accept secure, port 988 [ 1748.645442] Key type lgssc registered [ 1748.863659] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1750.835468] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1750.840140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1751.977208] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 1751.982705] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.56/24@tcp [ 1751.983971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1752.753374] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1755.067793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1755.110340] Lustre: 29547:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1755.177545] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1755.183287] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.56/24@tcp [ 1755.197334] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1756.007158] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1757.489742] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1758.393016] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1760.341708] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1760.351983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1760.432633] Lustre: 30659:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1760.478693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1761.792253] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1762.479526] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 1763.249313] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1772.497569] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1772.502691] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1772.510052] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1775.488894] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1775.491108] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1777.600412] LustreError: 31524:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1777.661723] Lustre: server umount lustre-OST0000 complete [ 1780.224947] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1780.228129] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1780.233589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1780.239023] Lustre: Skipped 1 previous similar message [ 1784.885332] LustreError: 31794:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1784.890473] LustreError: 31794:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1784.908198] Lustre: server umount lustre-MDT0000 complete [ 1785.504853] LustreError: 28685:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1785.510717] LustreError: 28685:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 1786.013327] LustreError: 30682:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748868932 with bad export cookie 8878649519912545806 [ 1786.014803] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1786.021955] LustreError: 30682:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1786.075588] LustreError: 32064:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1786.078155] LustreError: 32064:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1786.147343] Lustre: server umount lustre-MDT0001 complete [ 1789.478894] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 1790.125367] Key type lgssc unregistered [ 1790.221444] LNet: 32590:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1791.227605] LNet: Removed LNI 192.168.202.156@tcp [ 1797.640076] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 1801.937627] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 1802.049087] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1802.078813] alg: No test for adler32 (adler32-zlib) [ 1802.891635] Lustre: Lustre: Build Version: 2.16.55_39_g76ed203 [ 1802.943606] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 1802.945789] LNet: Accept secure, port 988 [ 1804.478490] Key type lgssc registered [ 1804.708687] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1806.953552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1808.905129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1810.855786] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1810.860284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1812.590014] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1812.596838] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1814.755171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1814.769084] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1814.774054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1815.840761] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1815.850580] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1815.884046] Lustre: lustre-MDT0000: new disk, initializing [ 1815.903979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1815.911051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1816.724916] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1820.450259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1820.475853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1820.507439] Lustre: 4210:0:(mgs_llog.c:1347:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1820.516863] Lustre: 5160:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1820.529935] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1820.532202] Lustre: Skipped 1 previous similar message [ 1820.565141] Lustre: lustre-MDT0001: new disk, initializing [ 1820.587030] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1820.598196] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1820.601822] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1821.460200] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1824.196642] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1825.152245] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1826.972701] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1826.977078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1826.999703] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1827.004489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1827.073586] Lustre: 6356:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1827.085511] Lustre: lustre-OST0000: new disk, initializing [ 1827.088729] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1827.111002] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1828.575440] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1830.020060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1830.024258] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1830.052013] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1831.165662] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1832.025713] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1832.082721] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1832.932701] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1832.987877] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1835.041474] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1835.041732] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1835.042118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1835.048663] LustreError: Skipped 1 previous similar message [ 1839.542022] LustreError: 7760:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1839.610531] Lustre: server umount lustre-OST0000 complete [ 1845.056924] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1845.062375] Lustre: Skipped 1 previous similar message [ 1845.063214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1845.063216] Lustre: Skipped 1 previous similar message [ 1846.780775] LustreError: 8030:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1846.782493] LustreError: 8030:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1846.798968] Lustre: server umount lustre-MDT0000 complete [ 1847.761144] LustreError: 5174:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748868993 with bad export cookie 10756228808949660941 [ 1847.762191] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1847.767910] LustreError: 5174:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1847.810175] LustreError: 8302:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1847.812749] LustreError: 8302:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1847.873650] Lustre: server umount lustre-MDT0001 complete [ 1850.090884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1850.229370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1850.981059] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1853.052701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1853.076708] Lustre: 9785:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1853.954449] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1855.249856] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1856.131437] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1857.925093] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1857.930496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1858.009799] Lustre: 10896:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1858.054336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1858.057540] Lustre: Skipped 1 previous similar message [ 1859.390561] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1860.765257] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1861.634273] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1861.683155] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1862.524256] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1863.597959] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 1 sec [ 1867.419298] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1867.422746] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1867.442684] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1867.446419] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1867.478097] Lustre: 12553:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1867.488213] Lustre: lustre-OST0001: new disk, initializing [ 1867.490712] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1867.508274] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1868.300359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1868.302739] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1868.319343] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1868.721479] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1871.234396] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 1871.287697] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1872.168439] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 1872.222003] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1874.005197] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800a9b1f800.ost_server_uuid 50 [ 1874.322878] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8800a9b1f800.ost_server_uuid in FULL state after 0 sec [ 1878.321375] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1878.321922] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1878.322404] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 1878.322406] Lustre: Skipped 1 previous similar message [ 1878.332530] LustreError: Skipped 1 previous similar message [ 1881.529627] LustreError: 14013:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 1881.532858] LustreError: 14013:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1881.547611] Lustre: server umount lustre-OST0001 complete [ 1883.088921] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1883.092084] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1883.098051] Lustre: Skipped 1 previous similar message [ 1883.100128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1883.102553] Lustre: Skipped 1 previous similar message [ 1883.329858] LustreError: 10902:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1883.337042] LustreError: 10902:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 1888.336856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1888.337743] LustreError: 10903:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1888.345272] Lustre: Skipped 2 previous similar messages [ 1888.579914] LustreError: 14283:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1888.581828] LustreError: 14283:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1888.634072] Lustre: server umount lustre-OST0000 complete [ 1893.200961] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1893.204445] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1893.209052] Lustre: Skipped 1 previous similar message [ 1895.734351] Lustre: server umount lustre-MDT0000 complete [ 1896.812458] LustreError: 9799:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748869042 with bad export cookie 10756228808949661809 [ 1896.813572] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1896.831739] LustreError: 9799:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1896.868313] LustreError: 14826:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 1896.869981] LustreError: 14826:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1896.934534] Lustre: server umount lustre-MDT0001 complete [ 1902.251960] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 1902.871365] Key type lgssc unregistered [ 1902.966999] LNet: 15953:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1903.972700] LNet: Removed LNI 192.168.202.156@tcp [ 1912.616852] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 1917.038256] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 1917.227138] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1917.231974] alg: No test for adler32 (adler32-zlib) [ 1918.056765] Lustre: Lustre: Build Version: 2.16.55_39_g76ed203 [ 1918.108357] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 1918.110344] LNet: Accept secure, port 988 [ 1919.640527] Key type lgssc registered [ 1919.898060] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1922.160827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1923.888910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1925.517874] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1925.525568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1927.194779] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1927.198373] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1929.027168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1929.041581] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1929.047132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1930.121003] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1930.130482] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1930.165779] Lustre: lustre-MDT0000: new disk, initializing [ 1930.186699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1930.192871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1931.037588] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1934.673189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1934.695451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1934.724381] Lustre: 20750:0:(mgs_llog.c:1347:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1934.736230] Lustre: 21688:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1934.750014] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1934.752113] Lustre: Skipped 1 previous similar message [ 1934.787284] Lustre: lustre-MDT0001: new disk, initializing [ 1934.811878] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1934.824056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1934.827774] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1935.664945] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1938.269710] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1939.121224] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1941.031504] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1941.035222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1941.061145] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1941.065297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1941.135478] Lustre: 22882:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1941.148538] Lustre: lustre-OST0000: new disk, initializing [ 1941.150925] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1941.175773] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1942.483109] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1944.368523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1944.372795] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1944.397635] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1945.112340] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1945.998002] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1946.061542] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1946.962836] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1947.021560] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1949.392809] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1949.393087] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1949.393426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1949.400987] LustreError: Skipped 1 previous similar message [ 1949.573320] LustreError: 24283:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1949.621425] Lustre: server umount lustre-OST0000 complete [ 1954.400896] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1954.404884] Lustre: Skipped 1 previous similar message [ 1954.406402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1954.408755] Lustre: Skipped 2 previous similar messages [ 1956.677431] LustreError: 24551:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1956.680072] LustreError: 24551:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1956.699140] Lustre: server umount lustre-MDT0000 complete [ 1957.678231] LustreError: 20743:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748869103 with bad export cookie 10965950415965895439 [ 1957.679501] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1957.687495] LustreError: 20743:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1957.735962] LustreError: 24821:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1957.739066] LustreError: 24821:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1957.797930] Lustre: server umount lustre-MDT0001 complete [ 1959.886150] Lustre: DEBUG MARKER: == conf-sanity test 47a: server restart does not lose lru_resize ========================================================== 08:58:26 (1748869106) [ 1963.618143] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 1965.178246] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 1967.362448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1968.992680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1970.462249] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1970.465508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1972.016032] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 1972.018976] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1973.733815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1973.752889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1973.813467] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1973.821931] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1973.854107] Lustre: lustre-MDT0000: new disk, initializing [ 1973.871339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1973.875818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1974.634881] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1978.064571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1978.080511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1978.104359] Lustre: 28539:0:(mgs_llog.c:1347:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1978.112879] Lustre: 29475:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1978.123710] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1978.125299] Lustre: Skipped 1 previous similar message [ 1978.156450] Lustre: lustre-MDT0001: new disk, initializing [ 1978.177909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1978.187318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1978.190869] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1978.950349] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1981.392147] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1982.186841] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1984.030525] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1984.033236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1984.048543] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1984.051218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1984.116180] Lustre: 30672:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1985.350547] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1987.809448] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1991.555802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1991.559063] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1991.578113] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1994.494562] Lustre: Failing over lustre-OST0000 [ 1994.515953] LustreError: 31551:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1994.517942] LustreError: 31551:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1994.564644] Lustre: server umount lustre-OST0000 complete [ 2006.521482] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2006.524493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2006.581277] Lustre: 32088:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2006.593026] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2006.593432] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 2006.597919] Lustre: Skipped 2 previous similar messages [ 2006.613471] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2006.615905] Lustre: Skipped 1 previous similar message [ 2006.619335] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2007.787331] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2008.762303] Lustre: Failing over lustre-MDT0000 [ 2008.817509] LustreError: 32744:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2008.819838] LustreError: 32744:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2008.832657] Lustre: server umount lustre-MDT0000 complete [ 2008.952476] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2011.617008] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2011.621496] Lustre: Skipped 2 previous similar messages [ 2012.616388] Lustre: 18561:0:(client.c:2451:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1748869142/real 1748869142] req@ffff880098aa2a00 x1833822127707136/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1748869158 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2012.623300] Lustre: 18561:0:(client.c:2451:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2013.953613] LustreError: 28544:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2013.959378] LustreError: 28544:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 5 previous similar messages [ 2016.624739] LustreError: 29507:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2016.629110] LustreError: 29507:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 2017.625399] Lustre: 18559:0:(client.c:2451:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1748869147/real 1748869147] req@ffff880072830e00 x1833822127709184/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1748869163 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2018.961576] LustreError: 31428:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2020.991068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2021.024424] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2021.027988] Lustre: 28541:0:(mgc_request.c:1854:mgc_process_log()) MGC192.168.202.156@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2021.079490] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2021.099383] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2021.858542] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2023.969562] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2026.098211] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 2026.104515] Lustre: lustre-OST0000: Recovery over after 0:17, of 3 clients 3 recovered and 0 were evicted. [ 2026.108838] Lustre: Skipped 1 previous similar message [ 2026.150829] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2031.105324] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2031.105895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2031.109635] Lustre: Skipped 1 previous similar message [ 2032.794346] LustreError: 1394:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2032.796491] LustreError: 1394:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2032.843812] Lustre: server umount lustre-OST0000 complete [ 2036.113198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2036.115050] Lustre: Skipped 2 previous similar messages [ 2039.883565] Lustre: server umount lustre-MDT0000 complete [ 2040.868637] LustreError: 28530:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748869187 with bad export cookie 10965950415965897490 [ 2040.870798] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2040.875894] LustreError: 28530:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2041.001849] Lustre: server umount lustre-MDT0001 complete [ 2043.990605] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 2044.538714] Key type lgssc unregistered [ 2044.628670] LNet: 2460:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2045.633592] LNet: Removed LNI 192.168.202.156@tcp [ 2062.667028] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 2062.852579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2062.857659] alg: No test for adler32 (adler32-zlib) [ 2063.678570] Lustre: Lustre: Build Version: 2.16.55_39_g76ed203 [ 2063.735699] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 2063.738155] LNet: Accept secure, port 988 [ 2065.267428] Key type lgssc registered [ 2065.497992] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2067.523807] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2067.530826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2068.681468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2069.503032] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2071.926748] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2073.664344] LustreError: 4143:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2073.669062] LustreError: 4143:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2073.672583] LustreError: 4143:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 5, retries 0, failed: rc = -5 [ 2073.738525] LustreError: 4790:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2073.816703] Lustre: server umount lustre-MDT0000 complete [ 2077.574236] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 2079.326775] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 2081.604128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2083.171854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2084.592587] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2084.596165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2086.150949] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2086.155034] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2087.934901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2087.955263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2088.029519] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2088.039835] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2088.073883] Lustre: lustre-MDT0000: new disk, initializing [ 2088.092453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2088.099490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2088.900053] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2092.476399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2092.494558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2092.517323] Lustre: 8051:0:(mgs_llog.c:1347:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2092.528481] Lustre: 8992:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2092.538140] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2092.539617] Lustre: Skipped 1 previous similar message [ 2092.571456] Lustre: lustre-MDT0001: new disk, initializing [ 2092.596638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2092.607054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2092.610418] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2093.385947] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2095.879651] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2096.665484] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2098.347985] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2098.350789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2098.368897] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2098.371761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2098.438650] Lustre: 10187:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2098.450143] Lustre: lustre-OST0000: new disk, initializing [ 2098.451732] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2098.470441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2099.679789] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2102.352093] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2102.582276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2102.585208] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2102.607312] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2104.997538] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2106.364032] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 09:00:52 (1748869252) [ 2172.704716] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2172.704933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2172.708649] Lustre: Skipped 2 previous similar messages [ 2174.851734] LustreError: 12487:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2174.853853] LustreError: 12487:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2174.896369] Lustre: server umount lustre-MDT0000 complete [ 2175.862948] LustreError: 8044:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748869322 with bad export cookie 5951525358356255613 [ 2175.864546] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2175.867955] LustreError: 8044:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2177.712669] LustreError: 8056:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2177.712709] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2177.712819] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2177.712820] Lustre: Skipped 2 previous similar messages [ 2177.725466] LustreError: 8056:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 2180.720656] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2180.720660] LustreError: 8056:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2181.919020] LustreError: 12758:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2181.920652] LustreError: 12758:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2181.977054] Lustre: server umount lustre-MDT0001 complete [ 2188.959377] LustreError: 13029:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2188.961239] LustreError: 13029:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2189.012136] Lustre: server umount lustre-OST0000 complete [ 2191.378632] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 2193.110180] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 2195.322421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2197.022548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2198.544522] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2198.548080] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2200.095171] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2200.098236] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2201.840438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2201.857148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2201.920135] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2201.928705] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2201.959055] Lustre: lustre-MDT0000: new disk, initializing [ 2201.975400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2201.980842] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2202.811666] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2206.281496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2206.302222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2206.325622] Lustre: 15896:0:(mgs_llog.c:1347:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2206.334020] Lustre: 16832:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2206.343227] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2206.345487] Lustre: Skipped 1 previous similar message [ 2206.378028] Lustre: lustre-MDT0001: new disk, initializing [ 2206.406201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2206.410010] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2207.239254] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2209.842008] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2210.715085] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2212.567519] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2212.571529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2212.587574] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2212.591425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2212.658367] Lustre: 18028:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2212.684562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2212.687654] Lustre: Skipped 1 previous similar message [ 2213.957874] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2214.730935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2214.733096] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2214.746375] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2216.472564] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2217.374832] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2217.430752] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2218.265843] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2218.318158] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2219.745213] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2219.745327] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2219.745562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2219.751068] LustreError: Skipped 1 previous similar message [ 2224.753255] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2224.755316] Lustre: Skipped 2 previous similar messages [ 2224.811859] LustreError: 19429:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2224.813467] LustreError: 19429:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2224.864719] Lustre: server umount lustre-OST0000 complete [ 2226.417271] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2226.421315] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2226.425951] Lustre: Skipped 1 previous similar message [ 2231.943914] Lustre: server umount lustre-MDT0000 complete [ 2232.931005] LustreError: 16846:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748869379 with bad export cookie 5951525358356414429 [ 2232.932242] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2232.937011] LustreError: 16846:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2232.988207] LustreError: 19970:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2232.991186] LustreError: 19970:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 2233.054518] Lustre: server umount lustre-MDT0001 complete [ 2235.035700] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 09:03:01 (1748869381) [ 2238.581043] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 2240.358642] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 2242.574633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2244.128253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2245.709576] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2245.716171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2247.454483] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2247.459449] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2249.351557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2249.371712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2249.451266] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2249.464843] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2249.467318] Lustre: Skipped 1 previous similar message [ 2249.507260] Lustre: lustre-MDT0000: new disk, initializing [ 2249.509803] Lustre: Skipped 1 previous similar message [ 2249.540851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2249.549865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2250.443464] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2253.811797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2253.830264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2253.859407] Lustre: 23689:0:(mgs_llog.c:1347:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2253.868042] Lustre: 24627:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2253.927572] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2254.710627] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2257.367953] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2258.232284] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2260.059500] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2260.063213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2260.081516] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2260.084111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2261.520717] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2264.047816] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2268.658023] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2268.679797] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2273.681027] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2273.681254] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2273.681256] Lustre: Skipped 1 previous similar message [ 2273.681608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2273.681609] Lustre: Skipped 3 previous similar messages [ 2273.688891] LustreError: Skipped 1 previous similar message [ 2277.212357] LustreError: 26839:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2277.214408] LustreError: 26839:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2277.267934] Lustre: server umount lustre-OST0000 complete [ 2284.356116] Lustre: server umount lustre-MDT0000 complete [ 2285.419245] LustreError: 25853:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748869431 with bad export cookie 5951525358356415297 [ 2285.421144] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2285.424941] LustreError: 25853:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2285.548324] Lustre: server umount lustre-MDT0001 complete [ 2288.153552] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 09:03:54 (1748869434) [ 2292.326794] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 2294.208723] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 2296.480520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2298.223641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2299.850568] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2299.854457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2301.482821] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2301.485918] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2303.327762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2303.347798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2303.421007] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2303.433314] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2303.436066] Lustre: Skipped 3 previous similar messages [ 2303.473030] Lustre: lustre-MDT0000: new disk, initializing [ 2303.474394] Lustre: Skipped 2 previous similar messages [ 2303.495936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2303.502983] Lustre: Skipped 2 previous similar messages [ 2303.513102] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2303.521938] Lustre: Skipped 2 previous similar messages [ 2304.382905] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2307.915247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2307.937132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2307.965739] Lustre: 31097:0:(mgs_llog.c:1347:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2307.979619] Lustre: 32035:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2307.985056] Lustre: 32035:0:(mgc_request_server.c:550:mgc_llog_local_copy()) Skipped 1 previous similar message [ 2308.057722] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2308.899681] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2311.612134] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2312.503902] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2314.489465] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2314.493848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2314.513343] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2314.517839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2315.977663] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2318.563719] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2318.595890] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2323.584896] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2323.586799] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2323.589474] Lustre: Skipped 3 previous similar messages [ 2323.590902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2323.592163] Lustre: Skipped 5 previous similar messages [ 2326.008828] LustreError: 1810:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2326.011897] LustreError: 1810:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2326.067902] Lustre: server umount lustre-OST0000 complete [ 2328.080966] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2328.083929] LustreError: Skipped 1 previous similar message [ 2333.151177] Lustre: server umount lustre-MDT0000 complete [ 2333.616688] LustreError: 31106:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2333.621015] LustreError: 31106:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 2334.125990] LustreError: 2083:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748869480 with bad export cookie 5951525358356416368 [ 2334.127539] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2334.131907] LustreError: 2083:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2334.251017] Lustre: server umount lustre-MDT0001 complete [ 2337.491822] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 2338.171585] Key type lgssc unregistered [ 2338.265392] LNet: 2877:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2339.270654] LNet: Removed LNI 192.168.202.156@tcp [ 2345.585395] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 2345.754958] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2345.763833] alg: No test for adler32 (adler32-zlib) [ 2346.582067] Lustre: Lustre: Build Version: 2.16.55_39_g76ed203 [ 2346.627691] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 2346.629747] LNet: Accept secure, port 988 [ 2348.156410] Key type lgssc registered [ 2348.347852] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2350.123912] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2350.128380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2351.248178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2351.988593] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2354.181935] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2354.207736] Lustre: 5258:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2354.293122] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2355.107165] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2356.471271] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2357.308441] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2359.216854] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2359.220785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2359.282599] Lustre: 6368:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2359.330239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2360.744130] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2361.331644] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2362.115384] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2379.361071] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2379.361647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2379.366353] Lustre: Skipped 1 previous similar message [ 2379.367816] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2384.012716] LustreError: 7181:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2384.076636] Lustre: server umount lustre-OST0000 complete [ 2389.329083] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2389.332387] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2389.338142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2389.343972] Lustre: Skipped 1 previous similar message [ 2391.113218] LustreError: 7452:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2391.115416] LustreError: 7452:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2391.132633] Lustre: server umount lustre-MDT0000 complete [ 2392.173108] LustreError: 7072:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748869538 with bad export cookie 4001145882377983273 [ 2392.174687] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2392.181848] LustreError: 7072:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2392.237012] LustreError: 7723:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2392.241180] LustreError: 7723:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2392.306870] Lustre: server umount lustre-MDT0001 complete [ 2395.450064] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 2396.072126] Key type lgssc unregistered [ 2396.163929] LNet: 8250:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2397.169636] LNet: Removed LNI 192.168.202.156@tcp [ 2404.078384] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 2404.265170] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2404.271392] alg: No test for adler32 (adler32-zlib) [ 2405.096053] Lustre: Lustre: Build Version: 2.16.55_39_g76ed203 [ 2405.152390] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 2405.154609] LNet: Accept secure, port 988 [ 2406.685439] Key type lgssc registered [ 2406.908093] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2408.867245] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2408.872466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2410.010379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2410.841121] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2413.136267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2413.185914] Lustre: 10625:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2413.302749] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2414.117112] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2415.491485] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2416.407393] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2418.299597] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2418.304127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2418.370222] Lustre: 11735:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2418.413498] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2419.742821] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2420.415011] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 2421.155772] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2428.417034] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2428.417123] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2428.417481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2429.820352] LustreError: 12534:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2429.874330] Lustre: server umount lustre-OST0000 complete [ 2431.250244] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8800aa341000.ost_server_uuid 50 [ 2448.928743] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 2463.360851] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2463.363469] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2463.367905] Lustre: Skipped 1 previous similar message [ 2463.370499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2463.373064] Lustre: Skipped 1 previous similar message [ 2464.717924] LustreError: 13016:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2464.720057] LustreError: 13016:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2464.734956] Lustre: server umount lustre-MDT0000 complete [ 2465.802246] LustreError: 9746:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748869611 with bad export cookie 4592006209539473157 [ 2465.803711] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2465.811109] LustreError: 9746:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2465.861107] LustreError: 13289:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2465.863563] LustreError: 13289:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2465.930062] Lustre: server umount lustre-MDT0001 complete [ 2468.104459] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 09:06:54 (1748869614) [ 2469.856090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2469.984846] LustreError: 14167:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2469.991916] LustreError: 14167:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 2470.005720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2470.859001] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2473.264233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2473.296027] Lustre: 15032:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2474.186623] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2475.639394] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2476.473448] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2478.162894] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2478.166218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2478.220680] Lustre: 16141:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2478.252869] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2478.255160] Lustre: Skipped 1 previous similar message [ 2479.456682] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2480.252784] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 2480.777563] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2482.474995] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2482.479431] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2482.496424] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2482.503237] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2482.537624] Lustre: 17256:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2482.548407] Lustre: lustre-OST0001: new disk, initializing [ 2482.550826] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2482.788073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2482.790262] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2482.807255] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2483.795315] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2486.255127] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2488.801418] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2488.801712] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2488.801713] Lustre: Skipped 1 previous similar message [ 2488.802331] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2488.802332] Lustre: Skipped 1 previous similar message [ 2491.368460] Lustre: lustre-OST0000: Not available for connect from 192.168.202.56@tcp (stopping) [ 2491.374596] Lustre: Skipped 1 previous similar message [ 2492.922356] LustreError: 18133:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2492.925069] LustreError: 18133:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2492.934067] Lustre: server umount lustre-OST0000 complete [ 2493.808938] LustreError: 17342:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2493.815269] LustreError: 17342:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 2494.083326] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2494.129857] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2494.906437] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2494.949596] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 2496.369513] LustreError: 17351:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2498.817815] LustreError: 16148:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2498.824285] LustreError: 16148:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 2503.825092] LustreError: 16148:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2503.830436] LustreError: 16148:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 2507.840956] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2507.841051] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2507.841053] Lustre: Skipped 1 previous similar message [ 2507.841493] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2507.852668] LustreError: Skipped 1 previous similar message [ 2510.701700] LustreError: 18938:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 2510.705065] LustreError: 18938:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2510.792945] Lustre: server umount lustre-OST0001 complete [ 2513.424927] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2513.428571] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2513.434787] Lustre: Skipped 1 previous similar message [ 2518.009550] Lustre: server umount lustre-MDT0000 complete [ 2518.848795] LustreError: 14167:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2518.853604] LustreError: 14167:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 2519.019158] LustreError: 14148:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748869665 with bad export cookie 4592006209539474214 [ 2519.020489] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2519.028615] LustreError: 14148:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2519.081861] LustreError: 19480:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 2519.088023] LustreError: 19480:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2519.156153] Lustre: server umount lustre-MDT0001 complete [ 2520.635343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2521.644346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2522.686258] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2522.691207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2523.742266] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2523.747014] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2525.403151] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 09:07:51 (1748869671) [ 2527.284649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2527.385567] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2527.398230] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2527.472262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2527.474249] Lustre: Skipped 1 previous similar message [ 2528.397322] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2530.731958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2530.757449] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2530.767230] Lustre: 21434:0:(mgs_llog.c:1347:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2530.774965] Lustre: 22305:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2531.714065] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2533.158534] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2534.040565] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2535.959930] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2535.965038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2536.023690] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2537.358371] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2538.725067] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2540.656497] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2540.659861] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2540.677121] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2542.130197] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2543.268453] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 2543.507697] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2548.256830] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2548.258631] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2548.261570] Lustre: Skipped 1 previous similar message [ 2548.263218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2548.264422] Lustre: Skipped 6 previous similar messages [ 2550.213352] LustreError: 25360:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2550.215384] LustreError: 25360:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 2550.224894] Lustre: server umount lustre-OST0000 complete [ 2553.297172] LustreError: 23438:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2553.302104] LustreError: 23438:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 2566.676616] Lustre: server umount lustre-OST0001 complete [ 2568.320947] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2568.321247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2568.321248] Lustre: Skipped 4 previous similar messages [ 2568.327777] Lustre: Skipped 4 previous similar messages [ 2573.767719] Lustre: server umount lustre-MDT0000 complete [ 2574.826992] LustreError: 21426:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748869721 with bad export cookie 4592006209539475572 [ 2574.828468] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2574.836610] LustreError: 21426:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2574.941829] Lustre: server umount lustre-MDT0001 complete [ 2576.440717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2577.480176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2578.517781] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2578.522060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2579.606672] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2579.610246] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2581.255891] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 09:08:47 (1748869727) [ 2585.260512] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 2587.191306] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 2589.855924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2591.703199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2593.293269] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2593.297142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2594.996873] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2595.000428] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2596.932100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2596.950598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2597.022530] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2597.032603] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2597.070330] Lustre: lustre-MDT0000: new disk, initializing [ 2597.091156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2597.094444] Lustre: Skipped 3 previous similar messages [ 2597.100363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2598.008127] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2601.602332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2601.616001] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2601.641036] Lustre: 30982:0:(mgs_llog.c:1347:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2601.651917] Lustre: 31918:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2601.655476] Lustre: 31918:0:(mgc_request_server.c:550:mgc_llog_local_copy()) Skipped 2 previous similar messages [ 2601.664862] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2601.667281] Lustre: Skipped 1 previous similar message [ 2601.703219] Lustre: lustre-MDT0001: new disk, initializing [ 2601.731843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2601.736729] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2602.565358] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2605.207556] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2606.053132] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2607.853938] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2607.858630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2607.873597] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2607.878967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2607.958686] Lustre: lustre-OST0000: new disk, initializing [ 2607.964220] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2609.285411] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2611.340951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2611.343686] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2611.367246] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2611.918747] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2612.785563] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2612.838407] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2613.712934] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2613.764101] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2616.353069] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2616.355655] LustreError: Skipped 3 previous similar messages [ 2616.356571] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2616.360464] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2616.362113] Lustre: Skipped 3 previous similar messages [ 2620.290906] LustreError: 2086:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2620.292792] LustreError: 2086:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 2620.367267] Lustre: server umount lustre-OST0000 complete [ 2627.514639] Lustre: server umount lustre-MDT0000 complete [ 2628.530152] LustreError: 677:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748869774 with bad export cookie 4592006209539477021 [ 2628.531238] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2628.536543] LustreError: 677:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2628.651117] Lustre: server umount lustre-MDT0001 complete [ 2631.050975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2631.204193] LustreError: 3246:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2631.210255] LustreError: 3246:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 6 previous similar messages [ 2631.224143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2631.226835] Lustre: Skipped 2 previous similar messages [ 2632.110686] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2634.432671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2634.460949] Lustre: 4107:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2634.464078] Lustre: 4107:0:(mgc_request_server.c:550:mgc_llog_local_copy()) Skipped 1 previous similar message [ 2635.351144] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2636.737275] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2637.590083] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2639.371022] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2639.374134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2640.788048] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2642.174512] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2643.040933] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2643.095910] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2643.902604] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2644.963255] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 1 sec [ 2649.472983] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2649.475188] LustreError: Skipped 3 previous similar messages [ 2651.517486] Lustre: server umount lustre-OST0000 complete [ 2652.781271] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2652.825432] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2653.618907] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2653.667547] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 2676.492304] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2676.495161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2677.897093] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2679.279782] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2679.684513] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 2686.428847] LustreError: 8550:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2686.431264] LustreError: 8550:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 2686.484366] Lustre: server umount lustre-OST0000 complete [ 2687.527453] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2687.527703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2687.527705] Lustre: Skipped 7 previous similar messages [ 2687.535973] Lustre: Skipped 6 previous similar messages [ 2693.594236] Lustre: server umount lustre-MDT0000 complete [ 2694.616076] LustreError: 3229:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748869840 with bad export cookie 4592006209539477889 [ 2694.617937] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2694.624272] LustreError: 3229:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2694.736394] Lustre: server umount lustre-MDT0001 complete [ 2696.655651] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 09:10:42 (1748869842) [ 2698.357885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2698.482851] LustreError: 9992:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2698.489259] LustreError: 9992:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 2698.498044] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2698.500316] Lustre: Skipped 3 previous similar messages [ 2699.316825] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2701.596436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2701.629259] Lustre: 10856:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2701.632920] Lustre: 10856:0:(mgc_request_server.c:550:mgc_llog_local_copy()) Skipped 2 previous similar messages [ 2702.566041] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2704.025323] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2704.903859] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2706.769073] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2706.772323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2708.075194] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2709.370581] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2710.222963] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2710.284648] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2711.131756] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2711.178382] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2712.907371] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2712.910191] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2712.927273] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2712.929946] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2712.967000] Lustre: lustre-OST0001: new disk, initializing [ 2712.972510] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2714.304669] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2714.477745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2714.480959] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2714.494760] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2716.881931] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2717.748182] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2717.796919] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2718.585368] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2718.641599] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2719.489399] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2725.201805] Lustre: server umount lustre-OST0001 complete [ 2726.612220] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2726.666609] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2727.551866] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2727.602762] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 2750.588503] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2750.592160] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2751.899917] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2753.287393] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2753.678286] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 2760.212156] Lustre: server umount lustre-OST0001 complete [ 2777.490568] Lustre: server umount lustre-OST0000 complete [ 2784.608778] Lustre: server umount lustre-MDT0000 complete [ 2785.611141] LustreError: 13626:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748869931 with bad export cookie 4592006209539478820 [ 2785.612208] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2785.616972] LustreError: 13626:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2791.722218] Lustre: server umount lustre-MDT0001 complete [ 2793.222622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2794.275186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2795.307562] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2795.312411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2796.381430] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2796.386593] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2797.935538] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 09:12:24 (1748869944) [ 2799.714458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2799.808665] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2799.818112] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2800.704698] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2802.939955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2802.964495] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2802.972725] Lustre: 19771:0:(mgs_llog.c:1347:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2803.812307] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2805.095903] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2805.900987] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2807.643523] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2807.646553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2807.704918] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2808.971023] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2810.279134] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2813.359204] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2813.362244] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2814.644732] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2815.888115] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2819.290518] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 2820.073553] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2820.132801] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2820.990996] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2821.050186] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2821.901125] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800a82f4000.ost_server_uuid 50 [ 2822.273518] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8800a82f4000.ost_server_uuid in FULL state after 0 sec [ 2822.557660] Lustre: Permanently deactivating lustre-OST0001 [ 2822.563652] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 2823.134233] Lustre: Permanently reactivating lustre-OST0001 [ 2824.945265] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2824.945981] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2824.945983] Lustre: Skipped 17 previous similar messages [ 2824.956189] Lustre: Skipped 7 previous similar messages [ 2829.843314] LustreError: 24617:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 2829.844847] LustreError: 24617:0:(obd_class.h:479:obd_check_dev()) Skipped 47 previous similar messages [ 2829.853075] Lustre: server umount lustre-OST0001 complete [ 2829.953469] LustreError: 21949:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2829.958810] LustreError: 21949:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 24 previous similar messages [ 2836.886500] Lustre: server umount lustre-OST0000 complete [ 2843.941032] Lustre: server umount lustre-MDT0000 complete [ 2844.910106] LustreError: 19804:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748869991 with bad export cookie 4592006209539480052 [ 2844.911036] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2844.915371] LustreError: 19804:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2845.020722] Lustre: server umount lustre-MDT0001 complete [ 2846.452671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2847.462418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2848.489266] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2848.492105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2849.464336] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2849.467794] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2851.058170] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 09:13:17 (1748869997) [ 2851.308711] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2851.313156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2853.041867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2853.124928] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2853.137362] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2853.143163] Lustre: Skipped 5 previous similar messages [ 2853.200361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2853.202265] Lustre: Skipped 8 previous similar messages [ 2854.034811] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2856.290519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2856.318519] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2856.321638] Lustre: Skipped 1 previous similar message [ 2856.330756] Lustre: 27458:0:(mgs_llog.c:1347:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2856.339557] Lustre: 28329:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2856.345318] Lustre: 28329:0:(mgc_request_server.c:550:mgc_llog_local_copy()) Skipped 6 previous similar messages [ 2857.215470] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2858.499897] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2859.290610] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2861.014092] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2861.017264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2861.037661] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2861.040792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2861.101387] Lustre: Permanently deactivating lustre-OST0000 [ 2862.436240] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2863.697404] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2865.478895] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2865.482523] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2866.385487] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2866.411007] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 2866.884776] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2868.244524] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2868.644867] Lustre: Permanently reactivating lustre-OST0000 [ 2868.651016] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 2868.654002] Lustre: Skipped 3 previous similar messages [ 2876.294671] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 2876.297434] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2876.301847] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to (at 0@lo) [ 2876.311118] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 2876.315748] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 2881.424955] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2881.427876] LustreError: Skipped 6 previous similar messages [ 2885.996882] Lustre: server umount lustre-OST0001 complete [ 2893.063686] Lustre: server umount lustre-OST0000 complete [ 2896.082911] Lustre: server umount lustre-MDT0000 complete [ 2897.116994] LustreError: 27450:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748870043 with bad export cookie 4592006209539481613 [ 2897.118191] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2897.123457] LustreError: 27450:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2897.235594] Lustre: server umount lustre-MDT0001 complete [ 2900.178232] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 2900.725092] Key type lgssc unregistered [ 2900.816689] LNet: 330:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2901.821586] LNet: Removed LNI 192.168.202.156@tcp [ 2907.886212] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 2908.048250] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2908.052037] alg: No test for adler32 (adler32-zlib) [ 2908.872590] Lustre: Lustre: Build Version: 2.16.55_39_g76ed203 [ 2908.926986] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 2908.928458] LNet: Accept secure, port 988 [ 2910.458495] Key type lgssc registered [ 2910.673871] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2911.468258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2913.219173] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2913.224406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2914.387344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2915.151164] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2917.348259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2917.367071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2917.397534] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2917.406470] Lustre: 1938:0:(mgs_llog.c:1347:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2917.410824] Lustre: Permanently deactivating lustre-MDT0001 [ 2917.414858] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 2917.427902] Lustre: 2808:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2917.507560] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2918.248531] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2919.481997] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2920.252862] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2920.685664] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 2920.688330] LustreError: 1976:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2920.691773] LustreError: 1976:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2920.696052] LustreError: 1976:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 1, failed: rc = -108 [ 2921.935819] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2921.939047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2922.000121] Lustre: 3937:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2922.034298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2923.264593] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2924.031638] LustreError: 3944:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2924.034348] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 2924.043119] LustreError: 3944:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 2924.609763] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2926.418912] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2926.421834] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2926.442434] Lustre: 5069:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2926.480832] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2927.481928] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 2927.482422] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 2927.483618] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 2927.726345] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2929.064178] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2930.513478] Lustre: Permanently reactivating lustre-MDT0001 [ 2930.518864] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 2930.521049] Lustre: Skipped 2 previous similar messages [ 2936.962276] Lustre: lustre-MDT0001: Client 62d63504-e7fe-4f8e-adf0-72b6956be4ac (at 192.168.202.56@tcp) reconnecting [ 2938.496019] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2938.885086] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2940.027951] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 2940.083910] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 2940.940136] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 2940.996856] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 2941.676683] Lustre: Permanently deactivating lustre-MDT0001 [ 2941.684478] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 2941.686337] Lustre: Skipped 2 previous similar messages [ 2942.755200] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 2946.418074] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 2946.420557] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2946.425225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2947.505249] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2947.505376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2947.510855] Lustre: Skipped 2 previous similar messages [ 2950.822906] LustreError: 6928:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 2950.841073] Lustre: server umount lustre-MDT0000 complete [ 2951.820156] LustreError: 2827:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748870098 with bad export cookie 17776500686207209534 [ 2951.821175] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2951.826378] LustreError: 2827:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2951.877036] LustreError: 7200:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 2951.879768] LustreError: 7200:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2951.960520] Lustre: server umount lustre-MDT0001 complete [ 2958.998363] LustreError: 7465:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 2959.000224] LustreError: 7465:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2959.007447] Lustre: server umount lustre-OST0000 complete [ 2961.893862] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 09:15:08 (1748870108) [ 2964.567719] LustreError: 8800:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2964.569479] LustreError: 8800:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2964.633082] Lustre: server umount lustre-OST0001 complete [ 2965.941892] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 2967.690197] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 2969.990439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2971.696215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2973.225077] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2973.228596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2974.787924] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2974.792398] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2976.546745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2976.565548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2976.640919] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2976.644083] Lustre: Skipped 2 previous similar messages [ 2976.650302] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2976.685263] Lustre: lustre-MDT0000: new disk, initializing [ 2976.705985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2976.711605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2977.501230] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2981.060037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2981.082537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2981.111253] Lustre: 11461:0:(mgs_llog.c:1347:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2981.121861] Lustre: 12398:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2981.133555] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2981.135276] Lustre: Skipped 1 previous similar message [ 2981.174360] Lustre: lustre-MDT0001: new disk, initializing [ 2981.205432] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2981.209861] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2982.032813] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2984.588832] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2985.389967] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2987.115352] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2987.118305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2987.130257] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2987.132947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2987.209086] Lustre: 13598:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2987.218575] Lustre: lustre-OST0000: new disk, initializing [ 2987.220117] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2987.237524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2987.239628] Lustre: Skipped 1 previous similar message [ 2988.462743] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2989.181061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2989.184830] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2989.204887] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2990.998784] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2991.448312] LustreError: 11466:0:(mdt_handler.c:3246:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 2995.200480] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2995.204265] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2995.223010] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2995.226141] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2995.260108] Lustre: 14867:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2995.272042] Lustre: lustre-OST0001: new disk, initializing [ 2995.275090] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2996.629563] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2997.372791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2997.376074] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2997.393131] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2999.320310] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2999.730217] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3001.451418] LustreError: 11466:0:(mdt_handler.c:3246:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3001.457669] LustreError: 11466:0:(mdt_handler.c:3246:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3011.460436] LustreError: 11466:0:(mdt_handler.c:3246:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3012.417145] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3012.417232] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3012.417982] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3012.417984] Lustre: Skipped 3 previous similar messages [ 3012.424458] LustreError: Skipped 1 previous similar message [ 3016.407259] Lustre: lustre-OST0001: Not available for connect from 192.168.202.56@tcp (stopping) [ 3016.409084] Lustre: Skipped 1 previous similar message [ 3017.995796] LustreError: 15805:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3017.997493] LustreError: 15805:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3018.009175] Lustre: server umount lustre-OST0001 complete [ 3022.433904] LustreError: 13607:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3022.438977] LustreError: 13607:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 3027.441365] LustreError: 13608:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3027.446294] LustreError: 13608:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 3032.449366] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3032.449743] LustreError: 13607:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3032.450053] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3032.450054] Lustre: Skipped 2 previous similar messages [ 3032.460909] Lustre: Skipped 2 previous similar messages [ 3035.340480] LustreError: 16083:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3035.342420] LustreError: 16083:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3035.405273] Lustre: server umount lustre-OST0000 complete [ 3037.457116] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3042.462307] Lustre: server umount lustre-MDT0000 complete [ 3042.464753] LustreError: 11469:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3042.471374] LustreError: 11469:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3043.425090] LustreError: 11454:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748870189 with bad export cookie 17776500686207212341 [ 3043.425921] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3043.432970] LustreError: 11454:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3049.547119] Lustre: server umount lustre-MDT0001 complete [ 3052.530209] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 3053.114001] Key type lgssc unregistered [ 3053.202755] LNet: 17153:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3054.208592] LNet: Removed LNI 192.168.202.156@tcp [ 3062.706212] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 3067.137342] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 3067.262296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3067.291463] alg: No test for adler32 (adler32-zlib) [ 3068.108800] Lustre: Lustre: Build Version: 2.16.55_39_g76ed203 [ 3068.161378] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 3068.163381] LNet: Accept secure, port 988 [ 3069.694456] Key type lgssc registered [ 3069.933020] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3072.183181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3073.938582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3075.489139] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3075.492954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3077.043446] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3077.047444] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3078.835306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3078.851838] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3078.856325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3079.932887] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3079.945117] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3079.981077] Lustre: lustre-MDT0000: new disk, initializing [ 3080.000102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3080.007125] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3080.849081] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3084.257466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3084.271869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3084.295491] Lustre: 21959:0:(mgs_llog.c:1347:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3084.302711] Lustre: 22897:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3084.313960] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3084.315812] Lustre: Skipped 1 previous similar message [ 3084.350513] Lustre: lustre-MDT0001: new disk, initializing [ 3084.371517] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3084.380892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3084.385081] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3085.123479] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3087.608308] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3088.427015] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3090.252755] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3090.255963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3090.274340] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3090.277197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3090.346401] Lustre: 24095:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3090.357938] Lustre: lustre-OST0000: new disk, initializing [ 3090.359574] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3090.378203] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3091.667426] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3094.193192] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3095.032666] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3099.198336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3099.200681] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3099.218026] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3100.171527] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec [ 3100.991371] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3101.038801] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3104.225181] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3104.225252] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3104.225703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3104.232466] LustreError: Skipped 1 previous similar message [ 3107.538345] LustreError: 25536:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3107.595996] Lustre: server umount lustre-OST0000 complete [ 3109.232909] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3109.233227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3109.233228] Lustre: Skipped 1 previous similar message [ 3109.239197] Lustre: Skipped 2 previous similar messages [ 3114.240854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3114.243712] Lustre: Skipped 2 previous similar messages [ 3114.636775] LustreError: 25806:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3114.639403] LustreError: 25806:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3114.662838] Lustre: server umount lustre-MDT0000 complete [ 3115.621236] LustreError: 21952:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748870261 with bad export cookie 6917411774642987003 [ 3115.622504] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3115.632736] LustreError: 21952:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3115.682676] LustreError: 26076:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3115.687844] LustreError: 26076:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3115.757622] Lustre: server umount lustre-MDT0001 complete [ 3117.918654] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 09:17:44 (1748870264) [ 3119.611192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3119.743727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3120.516612] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3122.725300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3122.755343] Lustre: 27819:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3123.613758] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3124.891923] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3125.690632] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3127.379476] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3127.382341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3127.440635] Lustre: 28930:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3127.473519] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3127.477053] Lustre: Skipped 1 previous similar message [ 3128.666782] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3129.903420] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3173.537406] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3173.537415] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3173.539485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3173.545894] Lustre: Skipped 1 previous similar message [ 3176.922464] LustreError: 29823:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3176.924296] LustreError: 29823:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3176.997849] Lustre: server umount lustre-OST0000 complete [ 3178.155528] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3178.158675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 3181.413501] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3181.416853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3181.477173] Lustre: 31007:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3181.517135] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3182.804415] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3184.098662] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3193.516494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3193.521438] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3193.521663] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.156@tcp (at 0@lo) [ 3218.561523] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3218.561645] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3218.562165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3218.562167] Lustre: Skipped 1 previous similar message [ 3222.120355] LustreError: 31956:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3222.122366] LustreError: 31956:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3222.186584] Lustre: server umount lustre-OST0000 complete [ 3223.568922] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3223.571993] Lustre: Skipped 1 previous similar message [ 3227.984957] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3227.987666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3227.989301] Lustre: Skipped 2 previous similar messages [ 3229.211345] Lustre: server umount lustre-MDT0000 complete [ 3230.178167] LustreError: 26938:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748870376 with bad export cookie 6917411774642987871 [ 3230.179226] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3230.186204] LustreError: 26938:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3230.237228] LustreError: 32497:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3230.239202] LustreError: 32497:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 3230.301273] Lustre: server umount lustre-MDT0001 complete [ 3233.283928] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 3233.813850] Key type lgssc unregistered [ 3233.906647] LNet: 558:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3234.911618] LNet: Removed LNI 192.168.202.156@tcp [ 3241.507515] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 3241.659349] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3241.663388] alg: No test for adler32 (adler32-zlib) [ 3242.467687] Lustre: Lustre: Build Version: 2.16.55_39_g76ed203 [ 3242.513452] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 3242.515441] LNet: Accept secure, port 988 [ 3244.042435] Key type lgssc registered [ 3244.254677] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3246.081403] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3246.087531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3247.216794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3247.966519] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3250.203321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3250.252089] Lustre: 2959:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3250.337928] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3251.115645] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3252.340082] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3253.137364] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3254.895019] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3254.899210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3254.986131] Lustre: 4069:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3255.030840] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3256.240394] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3257.031248] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3257.514499] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3269.411159] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3269.415873] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3269.423893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3270.049200] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3270.049347] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3270.052692] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3274.367419] LustreError: 5909:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3274.453402] Lustre: server umount lustre-OST0000 complete [ 3280.064954] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3280.065356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3280.065357] Lustre: Skipped 1 previous similar message [ 3280.071201] Lustre: Skipped 1 previous similar message [ 3281.792467] LustreError: 6180:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3281.795378] LustreError: 6180:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3281.807795] Lustre: server umount lustre-MDT0000 complete [ 3282.712080] LustreError: 6181:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748870428 with bad export cookie 8264753367918371021 [ 3282.713555] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3282.717485] LustreError: 6181:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3282.820727] Lustre: server umount lustre-MDT0001 complete [ 3285.900368] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 3286.411100] Key type lgssc unregistered [ 3286.497663] LNet: 6977:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3287.502743] LNet: Removed LNI 192.168.202.156@tcp [ 3292.580513] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 3292.688752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3292.692830] alg: No test for adler32 (adler32-zlib) [ 3293.504715] Lustre: Lustre: Build Version: 2.16.55_39_g76ed203 [ 3293.555019] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 3293.556468] LNet: Accept secure, port 988 [ 3295.087388] Key type lgssc registered [ 3295.308567] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3297.226753] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3297.231892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3298.393267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3299.132425] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3301.392139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3301.419233] Lustre: 9087:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3301.501520] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3302.293192] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3303.616328] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3304.412676] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3306.132382] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3306.135386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3306.192157] Lustre: 10200:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3306.225018] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3307.417703] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3308.228420] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3308.608765] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3316.225207] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3316.225573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3316.230353] Lustre: Skipped 1 previous similar message [ 3318.165351] LustreError: 11196:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3318.215423] Lustre: server umount lustre-OST0000 complete [ 3321.221269] LustreError: 11464:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3321.222936] LustreError: 11464:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3321.232854] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3321.236980] LustreError: 9096:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3321.241712] LustreError: 9096:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 3321.247123] Lustre: server umount lustre-MDT0000 complete [ 3322.201088] LustreError: 8211:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748870468 with bad export cookie 3724036465305879547 [ 3322.202390] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3322.207692] LustreError: 8211:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3322.249145] LustreError: 11732:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3322.250832] LustreError: 11732:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3322.324246] Lustre: server umount lustre-MDT0001 complete [ 3325.374187] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 3326.027846] Key type lgssc unregistered [ 3326.136856] LNet: 12256:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3327.145709] LNet: Removed LNI 192.168.202.156@tcp [ 3334.298615] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 3334.411181] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3334.438326] alg: No test for adler32 (adler32-zlib) [ 3335.260799] Lustre: Lustre: Build Version: 2.16.55_39_g76ed203 [ 3335.328899] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 3335.332167] LNet: Accept secure, port 988 [ 3336.868521] Key type lgssc registered [ 3337.078760] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3338.943679] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3338.948658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3340.107179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3341.063839] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3343.347118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3343.381566] Lustre: 14628:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3343.464427] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3344.251856] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3345.553861] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3346.321234] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3348.026167] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3348.029753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3348.105509] Lustre: 15735:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3348.146513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3349.365532] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3350.148344] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3350.678073] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3363.168927] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3363.169259] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3363.169880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3368.177349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3368.179429] Lustre: Skipped 2 previous similar messages [ 3368.759592] LustreError: 17566:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3368.814735] Lustre: server umount lustre-OST0000 complete [ 3373.184944] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3373.187958] Lustre: Skipped 1 previous similar message [ 3373.188364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3375.933497] LustreError: 17837:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3375.935454] LustreError: 17837:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3375.947970] Lustre: server umount lustre-MDT0000 complete [ 3376.887163] LustreError: 13752:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748870523 with bad export cookie 6755362724909802095 [ 3376.888209] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3376.892987] LustreError: 13752:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3376.937964] LustreError: 18108:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3376.940815] LustreError: 18108:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3377.015822] Lustre: server umount lustre-MDT0001 complete [ 3379.924922] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 3380.464422] Key type lgssc unregistered [ 3380.556666] LNet: 18634:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3381.561606] LNet: Removed LNI 192.168.202.156@tcp [ 3386.354136] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 3386.489633] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3386.500215] alg: No test for adler32 (adler32-zlib) [ 3387.309129] Lustre: Lustre: Build Version: 2.16.55_39_g76ed203 [ 3387.356006] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 3387.357487] LNet: Accept secure, port 988 [ 3388.886429] Key type lgssc registered [ 3389.086950] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3390.974001] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3390.978883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3392.125403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3392.869424] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3395.046484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3395.074425] Lustre: 20783:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3395.151903] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3395.938386] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3397.183065] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3397.970298] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3399.631956] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3399.635174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3399.695820] Lustre: 21892:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3399.725758] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3400.928612] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3402.143119] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3402.726997] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 3412.753009] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3412.755218] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3412.759498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3415.745377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3415.745570] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3415.745584] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3415.754457] Lustre: Skipped 1 previous similar message [ 3417.040997] LustreError: 22883:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3417.106150] Lustre: server umount lustre-OST0000 complete [ 3420.177160] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3420.179121] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3420.183975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3424.136611] LustreError: 23152:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3424.138226] LustreError: 23152:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3424.151072] Lustre: server umount lustre-MDT0000 complete [ 3425.098320] LustreError: 20797:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748870571 with bad export cookie 8742431664739165839 [ 3425.099545] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3425.104816] LustreError: 20797:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3425.153811] LustreError: 23423:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3425.155485] LustreError: 23423:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3425.231481] Lustre: server umount lustre-MDT0001 complete [ 3428.283117] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 3428.798060] Key type lgssc unregistered [ 3428.878679] LNet: 23947:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3429.883540] LNet: Removed LNI 192.168.202.156@tcp [ 3432.366979] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 3436.343645] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 3440.680268] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 3440.795195] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3440.817399] alg: No test for adler32 (adler32-zlib) [ 3441.623726] Lustre: Lustre: Build Version: 2.16.55_39_g76ed203 [ 3441.668337] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 3441.670108] LNet: Accept secure, port 988 [ 3443.195496] Key type lgssc registered [ 3443.392588] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3445.448432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3447.086620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3448.559833] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3448.562781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3450.073999] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3450.076812] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3451.892865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3451.912252] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3451.915405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3452.983980] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3452.991442] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3453.032143] Lustre: lustre-MDT0000: new disk, initializing [ 3453.051309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3453.067958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3453.806519] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3457.155380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3457.173808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3457.201890] Lustre: 28112:0:(mgs_llog.c:1347:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3457.210490] Lustre: 29051:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3457.220158] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3457.222000] Lustre: Skipped 1 previous similar message [ 3457.256853] Lustre: lustre-MDT0001: new disk, initializing [ 3457.277973] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3457.286848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3457.289436] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3458.086799] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3460.667140] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3461.424801] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3463.127189] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3463.130512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3463.148309] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3463.150800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3463.222362] Lustre: 30249:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3463.233379] Lustre: lustre-OST0000: new disk, initializing [ 3463.234622] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3463.253458] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3464.461785] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3466.921459] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3467.705714] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3470.129248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3470.131785] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3470.149058] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3470.792274] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 3471.563833] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3471.609745] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3475.153079] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3475.153191] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3475.153507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3475.158803] LustreError: Skipped 1 previous similar message [ 3478.114402] LustreError: 31671:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3478.181609] Lustre: server umount lustre-OST0000 complete [ 3480.160947] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3480.161237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3480.161238] Lustre: Skipped 1 previous similar message [ 3480.168379] Lustre: Skipped 2 previous similar messages [ 3485.168842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3485.171302] Lustre: Skipped 2 previous similar messages [ 3485.196250] LustreError: 31941:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3485.197521] LustreError: 31941:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3485.211378] Lustre: server umount lustre-MDT0000 complete [ 3486.155989] LustreError: 30273:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748870632 with bad export cookie 6236088710706622023 [ 3486.157445] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3486.160889] LustreError: 30273:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3486.224235] LustreError: 32212:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3486.226219] LustreError: 32212:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3486.297985] Lustre: server umount lustre-MDT0001 complete [ 3488.584552] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 09:23:54 (1748870634) [ 3490.215871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3490.369179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3491.092664] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3493.185519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3493.212712] Lustre: 1522:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3494.008560] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3495.227735] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3496.018191] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3497.736975] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3497.740431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3497.811757] Lustre: 2629:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3497.846077] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3497.848329] Lustre: Skipped 1 previous similar message [ 3499.062486] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3500.318016] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3507.857259] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3507.857395] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3507.857711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3507.863703] Lustre: Skipped 1 previous similar message [ 3509.333617] LustreError: 3430:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3509.335569] LustreError: 3430:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3509.395010] Lustre: server umount lustre-OST0000 complete [ 3512.417978] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 3512.421141] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3512.426780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3512.429592] Lustre: Skipped 1 previous similar message [ 3516.436200] LustreError: 3698:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3516.437747] LustreError: 3698:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3516.455687] Lustre: server umount lustre-MDT0000 complete [ 3517.428003] LustreError: 613:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748870663 with bad export cookie 6236088710706622891 [ 3517.429380] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3517.429689] LustreError: 1537:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 3517.436252] LustreError: 613:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3517.555398] Lustre: server umount lustre-MDT0001 complete [ 3520.602376] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 3521.148199] Key type lgssc unregistered [ 3521.233695] LNet: 4492:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3522.239662] LNet: Removed LNI 192.168.202.156@tcp [ 3528.528616] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 3532.865045] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 3533.015205] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3533.019049] alg: No test for adler32 (adler32-zlib) [ 3533.823626] Lustre: Lustre: Build Version: 2.16.55_39_g76ed203 [ 3533.867780] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 3533.869482] LNet: Accept secure, port 988 [ 3535.394457] Key type lgssc registered [ 3535.582920] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3537.618650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3539.195766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3540.686545] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3540.689026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3542.217613] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3542.220147] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3543.448960] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3543.451680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3544.779137] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3544.781867] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3546.568048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3546.583649] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3546.588119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3547.673190] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3547.686015] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3547.730194] Lustre: lustre-MDT0000: new disk, initializing [ 3547.748891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3547.755274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3548.545413] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3552.150525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3552.168180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3552.193843] Lustre: 9111:0:(mgs_llog.c:1347:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3552.201797] Lustre: 10051:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3552.212241] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3552.214876] Lustre: Skipped 1 previous similar message [ 3552.251179] Lustre: lustre-MDT0001: new disk, initializing [ 3552.273786] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3552.286719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3552.290319] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3553.067404] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3555.586552] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3556.357824] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3558.129238] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3558.132224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3558.152738] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3558.155238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3558.232853] Lustre: 11252:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3558.243032] Lustre: lustre-OST2710: new disk, initializing [ 3558.244722] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 3558.263623] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 3559.469319] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3561.933034] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 3563.640439] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3563.643263] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3563.663557] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3563.666228] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3563.695353] Lustre: 12441:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3563.705686] Lustre: lustre-OST03e8: new disk, initializing [ 3563.708373] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 3563.715313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 3563.717582] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 3563.730456] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 3563.747728] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 3564.926992] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3566.153296] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 3568.757418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 3568.761239] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 3568.773863] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 3569.112076] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 3569.162119] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3569.940502] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 3569.990790] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3570.756341] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 3570.801061] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3571.564040] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 3571.610840] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3571.993208] Lustre: DEBUG MARKER: conf-sanity test_56a: @@@@@@ FAIL: Stripe count not two: 1 [ 3574.192768] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 09:25:20 (1748870720) [ 3574.523473] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 3574.904500] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 09:25:21 (1748870721) [ 3575.410523] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 3577.312803] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3577.315498] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3577.319855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3578.267878] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.56@tcp (stopping) [ 3578.768980] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3582.695661] LustreError: 15249:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 3582.715076] Lustre: server umount lustre-MDT0000 complete [ 3583.432633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3583.931816] LustreError: 10068:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748870730 with bad export cookie 12404788526285323271 [ 3583.932793] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3583.938387] LustreError: 10068:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3588.273594] LustreError: 9115:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3588.277794] LustreError: 9115:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 6 previous similar messages [ 3588.279618] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.56@tcp (stopping) [ 3588.279620] Lustre: Skipped 4 previous similar messages [ 3588.784781] Lustre: lustre-MDT0001-lwp-OST2710: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3588.784808] LustreError: 14436:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3588.796649] Lustre: Skipped 3 previous similar messages [ 3589.979655] LustreError: 15590:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 3589.981497] LustreError: 15590:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3590.047272] Lustre: server umount lustre-MDT0001 complete [ 3590.788718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3593.285757] Lustre: lustre-OST2710: Not available for connect from 192.168.202.56@tcp (stopping) [ 3593.287742] Lustre: Skipped 2 previous similar messages [ 3597.287117] LustreError: 15931:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3597.289090] LustreError: 15931:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3597.299480] Lustre: server umount lustre-OST2710 complete [ 3598.025973] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3598.029319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3598.289142] LustreError: 11261:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST2710: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3598.293570] LustreError: 11261:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 3598.634905] Lustre: server umount lustre-OST03e8 complete [ 3599.405088] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3599.411115] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3599.712930] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3599.716619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3601.607689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3601.729673] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3601.741940] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3601.810922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3602.658547] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3604.841738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3604.862974] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3604.871585] Lustre: 17028:0:(mgs_llog.c:1347:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3604.878628] Lustre: 17896:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3605.695713] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3607.011985] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5108.073840] Lustre: DEBUG MARKER: rpc test_57a: @@@@@@ FAIL: can't put import for mdc.lustre-MDT0000-mdc-*.mds_server_uuid into FULL state after 1475 sec, have DISCONN [ 5110.236368] Lustre: DEBUG MARKER: conf-sanity test_57a: @@@@@@ FAIL: import is not in FULL state [ 5111.885519] Lustre: DEBUG MARKER: == conf-sanity test 57b: initial registration from servicenode should not fail ========================================================== 09:50:57 (1748872257) [ 5112.445445] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 5114.336796] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5114.336968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5114.343203] Lustre: Skipped 1 previous similar message [ 5118.562790] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.56@tcp (stopping) [ 5118.564839] Lustre: Skipped 2 previous similar messages [ 5119.646351] LustreError: 20843:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 5119.650865] LustreError: 20843:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 5119.671817] Lustre: server umount lustre-MDT0000 complete [ 5120.686005] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5121.324083] LustreError: 17021:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748872267 with bad export cookie 12404788526285324776 [ 5121.326037] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5121.338137] LustreError: 17021:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5121.401343] LustreError: 21187:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 5121.405064] LustreError: 21187:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5121.473718] Lustre: server umount lustre-MDT0001 complete [ 5122.312094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5123.189497] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5123.193754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5124.512939] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 5124.516316] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5124.757547] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5124.761647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5126.924552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5127.063582] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5127.077079] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5127.173479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5127.178485] Lustre: Skipped 1 previous similar message [ 5128.363061] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5130.745493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5130.792311] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5130.801543] Lustre: 22484:0:(mgs_llog.c:1347:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5130.811053] Lustre: 23354:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.202.156@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5130.894780] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5131.904654] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5133.353436] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6641.988910] Lustre: DEBUG MARKER: rpc test_57b: @@@@@@ FAIL: can't put import for mdc.lustre-MDT0000-mdc-*.mds_server_uuid into FULL state after 1475 sec, have DISCONN