== sanity-quota test 91: new quota index files in quota_master ========================================================== 07:03:25 (1743505405) Stopping clients: oleg654-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg654-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg654-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg654-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg654-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg654-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg654-server oleg654-server: oleg654-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg654-server' oleg654-server: oleg654-server.virtnet: executing load_modules_local oleg654-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg654-server: Loading modules from /home/green/git/lustre-release/lustre oleg654-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg654-server: detected 4 online CPUs by sysfs oleg654-server: MODOPTS_LIBCFS= oleg654-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg654-server: oleg654-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg654-client: oleg654-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg654-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg654-server: oleg654-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg654-client: oleg654-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start ost2 service on oleg654-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg654-server: oleg654-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg654-client: oleg654-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid start client Starting client: oleg654-client.virtnet: -o user_xattr,flock 192.168.206.154@tcp:/lustre /mnt/lustre Creating new pool qpool1 oleg654-server: Pool lustre.qpool1 created Adding targets to pool oleg654-server: OST lustre-OST0000_UUID added to pool lustre.qpool1 oleg654-server: OST lustre-OST0001_UUID added to pool lustre.qpool1 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' stop mds1 Stopping /mnt/lustre-mds1 (opts:-f) on oleg654-server oleg654-server: debugfs 1.47.1-wc2 (08-Nov-2024) oleg654-server: debugfs 1.47.1-wc2 (08-Nov-2024) oleg654-server: debugfs 1.47.1-wc2 (08-Nov-2024) 32003 40755 (2) 0 0 4096 31-Dec-1969 19:00 . 32001 40755 (2) 0 0 4096 31-Dec-1969 19:00 .. 93 100644 (1) 0 0 8192 31-Dec-1969 19:00 0x20000 94 100644 (1) 0 0 8192 31-Dec-1969 19:00 0x1020000 95 100644 (1) 0 0 8192 31-Dec-1969 19:00 0x2020000 161 100644 (1) 0 0 8192 31-Dec-1969 19:00 0x20000-MDT0000_UUID start mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg654-server: oleg654-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg654-client: oleg654-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 10 sec running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d91.sanity-quota/f1] [bs=1M] [count=50] 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.10659 s, 47.4 MB/s Stopping clients: oleg654-client.virtnet /mnt/lustre (opts:) Stopping client oleg654-client.virtnet /mnt/lustre opts: Stopping clients: oleg654-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg654-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg654-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg654-server Stopping clients: oleg654-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg654-client.virtnet /mnt/lustre2 (opts:-f) oleg654-server: oleg654-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg654-server' oleg654-server: oleg654-server.virtnet: executing load_modules_local oleg654-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg654-server: Loading modules from /home/green/git/lustre-release/lustre oleg654-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg654-server: detected 4 online CPUs by sysfs oleg654-server: MODOPTS_LIBCFS= oleg654-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey Checking servers environments Checking clients oleg654-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg654-server' oleg654-server: oleg654-server.virtnet: executing load_modules_local oleg654-server: Loading modules from /home/green/git/lustre-release/lustre oleg654-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg654-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg654-server: detected 4 online CPUs by sysfs oleg654-server: MODOPTS_LIBCFS= oleg654-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg654-server: oleg654-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg654-client: oleg654-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg654-server: oleg654-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg654-client: oleg654-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg654-server: oleg654-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg654-client: oleg654-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg654-server: oleg654-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg654-client: oleg654-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 Starting client: oleg654-client.virtnet: -o user_xattr,flock 192.168.206.154@tcp:/lustre /mnt/lustre Starting client oleg654-client.virtnet: -o user_xattr,flock 192.168.206.154@tcp:/lustre /mnt/lustre Started clients oleg654-client.virtnet: 192.168.206.154@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt,statfs_project) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffffa0440669c000.idle_timeout=debug osc.lustre-OST0001-osc-ffffa0440669c000.idle_timeout=debug Destroy the created pools: qpool1