-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:36:53 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg133-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg133-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg133-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg133-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg133-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg133-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg133-server oleg133-server: oleg133-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing load_modules_local oleg133-server: Loading modules from /home/green/git/lustre-release/lustre oleg133-server: detected 4 online CPUs by sysfs oleg133-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg133-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg133-server: oleg133-server.virtnet: executing set_default_debug -1 all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg133-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg133-server: oleg133-server.virtnet: executing set_default_debug -1 all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg133-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg133-server: oleg133-server.virtnet: executing set_default_debug -1 all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg133-server: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg133-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg133-server: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg133-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg133-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg133-server stop mds service on oleg133-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg133-server stop mds service on oleg133-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg133-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:38:17 (1709786297) start mds service on oleg133-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg133-server: oleg133-server.virtnet: executing set_default_debug -1 all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg133-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg133-server: oleg133-server.virtnet: executing set_default_debug -1 all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg133-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg133-server: oleg133-server.virtnet: executing set_default_debug -1 all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Started lustre-OST0000 oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg133-client.virtnet: -o user_xattr,flock oleg133-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h38m36s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff88012a606000.stats=clear mdc.lustre-MDT0001-mdc-ffff88012a606000.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:39:20.000000000 -0500 Modify: 2024-03-06 23:39:20.000000000 -0500 Change: 2024-03-06 23:39:20.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 86 lines, 86 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg133-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg133-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg133-server stop mds service on oleg133-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg133-server stop mds service on oleg133-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg133-server unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local modules unloaded. oleg133-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg133-server: Is the ldiskfs module available? oleg133-server: oleg133-server: tunefs.lustre FATAL: failed to write local files oleg133-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg133-client: oleg133-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg133-server: tunefs.lustre: Unable to mount /dev/mapper/mds2_flakey: No such device oleg133-server: Is the ldiskfs module available? oleg133-server: oleg133-server: tunefs.lustre FATAL: failed to write local files oleg133-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg133-client: oleg133-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1 (MDT ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.201.133@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x101 (MDT writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.201.133@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg133-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg133-server: Is the ldiskfs module available? oleg133-server: oleg133-server: tunefs.lustre FATAL: failed to write local files oleg133-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg133-client: oleg133-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.133@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.133@tcp sys.timeout=20 oleg133-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg133-server: Is the ldiskfs module available? oleg133-server: oleg133-server: tunefs.lustre FATAL: failed to write local files oleg133-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg133-client: oleg133-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.133@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x162 (OST first_time update writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.133@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg133-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg133-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: oleg133-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing load_modules_local oleg133-server: Loading modules from /home/green/git/lustre-release/lustre oleg133-server: detected 4 online CPUs by sysfs oleg133-server: Force libcfs to create 2 CPU partitions oleg133-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg133-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg133-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg133-server: oleg133-server.virtnet: executing set_default_debug -1 all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg133-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg133-server: oleg133-server.virtnet: executing set_default_debug -1 all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg133-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg133-server: oleg133-server.virtnet: executing set_default_debug -1 all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg133-server: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg133-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec oleg133-server: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg133-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg133-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg133-server stop mds service on oleg133-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg133-server stop mds service on oleg133-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg133-server PASS 35b (143s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:40:42 (1709786442) start mds service on oleg133-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg133-server: oleg133-server.virtnet: executing set_default_debug -1 all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg133-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg133-server: oleg133-server.virtnet: executing set_default_debug -1 all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg133-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg133-server: oleg133-server.virtnet: executing set_default_debug -1 all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Started lustre-OST0000 oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg133-client.virtnet: -o user_xattr,flock oleg133-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg133-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=34 stop mds service on oleg133-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg133-server stop mds service on oleg133-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg133-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg133-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg133-server: oleg133-server.virtnet: executing set_default_debug -1 all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg133-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg133-server: oleg133-server.virtnet: executing set_default_debug -1 all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg133-client.virtnet: -o user_xattr,flock oleg133-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=66 umount lustre on /mnt/lustre..... Stopping client oleg133-client.virtnet /mnt/lustre (opts:) stop mds service on oleg133-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg133-server stop mds service on oleg133-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg133-server 000000 2 000008 oleg133-server: 8+0 records in oleg133-server: 8+0 records out oleg133-server: 4096 bytes (4.1 kB) copied, 0.000275808 s, 14.9 MB/s start mds service on oleg133-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg133-server: oleg133-server.virtnet: executing set_default_debug -1 all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg133-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg133-server: oleg133-server.virtnet: executing set_default_debug -1 all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg133-client.virtnet: -o user_xattr,flock oleg133-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=98 umount lustre on /mnt/lustre..... Stopping client oleg133-client.virtnet /mnt/lustre (opts:) stop mds service on oleg133-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg133-server stop mds service on oleg133-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg133-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg133-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg133-server stop mds service on oleg133-server stop mds service on oleg133-server unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg133-client: oleg133-client: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-client: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 38 (110s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:42:34 (1709786554) Stopping clients: oleg133-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg133-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: oleg133-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing load_modules_local oleg133-server: Loading modules from /home/green/git/lustre-release/lustre oleg133-server: detected 4 online CPUs by sysfs oleg133-server: Force libcfs to create 2 CPU partitions oleg133-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg133-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg133-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg133-server: oleg133-server.virtnet: executing set_default_debug -1 all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg133-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg133-server: oleg133-server.virtnet: executing set_default_debug -1 all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg133-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg133-server: oleg133-server.virtnet: executing set_default_debug -1 all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg133-client.virtnet: -o user_xattr,flock oleg133-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg133-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg133-server: oleg133-server.virtnet: executing set_default_debug -1 all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg133-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg133-server umount lustre on /mnt/lustre..... Stopping client oleg133-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg133-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg133-server stop mds service on oleg133-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg133-server stop mds service on oleg133-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg133-server unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local modules unloaded. oleg133-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg133-server: Is the ldiskfs module available? oleg133-server: oleg133-server: tunefs.lustre FATAL: failed to write local files oleg133-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg133-client: oleg133-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg133-server: tunefs.lustre: Unable to mount /dev/mapper/mds2_flakey: No such device oleg133-server: Is the ldiskfs module available? oleg133-server: oleg133-server: tunefs.lustre FATAL: failed to write local files oleg133-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg133-client: oleg133-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1 (MDT ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.201.133@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x101 (MDT writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.201.133@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg133-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg133-server: Is the ldiskfs module available? oleg133-server: oleg133-server: tunefs.lustre FATAL: failed to write local files oleg133-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg133-client: oleg133-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.133@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.133@tcp sys.timeout=20 oleg133-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg133-server: Is the ldiskfs module available? oleg133-server: oleg133-server: tunefs.lustre FATAL: failed to write local files oleg133-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg133-client: oleg133-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.133@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.133@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg133-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg133-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: oleg133-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing load_modules_local oleg133-server: Loading modules from /home/green/git/lustre-release/lustre oleg133-server: detected 4 online CPUs by sysfs oleg133-server: Force libcfs to create 2 CPU partitions oleg133-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg133-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg133-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg133-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg133-server: oleg133-server.virtnet: executing set_default_debug -1 all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg133-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg133-server: oleg133-server.virtnet: executing set_default_debug -1 all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg133-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg133-server: oleg133-server.virtnet: executing set_default_debug -1 all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg133-server: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg133-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg133-server: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg133-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg133-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg133-server stop mds service on oleg133-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg133-server stop mds service on oleg133-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg133-server PASS 51 (193s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:45:51 (1709786751) start mds service on oleg133-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg133-server: oleg133-server.virtnet: executing set_default_debug -1 all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg133-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg133-server: oleg133-server.virtnet: executing set_default_debug -1 all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg133-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg133-server: oleg133-server.virtnet: executing set_default_debug -1 all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Started lustre-OST0000 oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg133-client.virtnet: -o user_xattr,flock oleg133-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 222 (time 1709786798.79 total 10.57 last 21.00) - open/close 438 (time 1709786808.94 total 20.72 last 21.29) - open/close 809 (time 1709786819.96 total 31.73 last 33.68) - open/close 1105 (time 1709786830.93 total 42.71 last 26.98) - open/close 1359 (time 1709786840.93 total 52.71 last 25.39) - open/close 1671 (time 1709786850.94 total 62.72 last 31.19) - open/close 1736 (time 1709786861.03 total 72.80 last 6.44) - open/close 1948 (time 1709786871.19 total 82.97 last 20.86) - open/close 2166 (time 1709786881.20 total 92.98 last 21.78) - open/close 2422 (time 1709786901.84 total 113.62 last 12.40) - open/close 2669 (time 1709786913.94 total 125.71 last 20.43) - open/close 2822 (time 1709786924.97 total 136.74 last 13.87) - open/close 3149 (time 1709786935.20 total 146.98 last 31.95) - open/close 3465 (time 1709786945.43 total 157.21 last 30.89) - open/close 3651 (time 1709786956.05 total 167.82 last 17.51) - open/close 3779 (time 1709786966.05 total 177.83 last 12.79) - open/close 3956 (time 1709786976.42 total 188.19 last 17.08) - open/close 4235 (time 1709786986.85 total 198.62 last 26.75) - open/close 4652 (time 1709786996.85 total 208.63 last 41.68) - open/close 4894 (time 1709787007.67 total 219.45 last 22.37) - open/close 5044 (time 1709787019.70 total 231.47 last 12.47) - open/close 5290 (time 1709787030.17 total 241.95 last 23.49) - open/close 5506 (time 1709787040.33 total 252.10 last 21.27) - open/close 5753 (time 1709787051.11 total 262.89 last 22.91) - open/close 6077 (time 1709787062.15 total 273.92 last 29.36) - open/close 6286 (time 1709787072.32 total 284.10 last 20.54) - open/close 6479 (time 1709787082.51 total 294.28 last 18.95) - open/close 6664 (time 1709787092.53 total 304.30 last 18.46) - open/close 6905 (time 1709787103.02 total 314.80 last 22.96) - open/close 7152 (time 1709787113.80 total 325.57 last 22.92) - open/close 7271 (time 1709787123.81 total 335.58 last 11.89) - open/close 7291 (time 1709787135.25 total 347.03 last 1.75) - open/close 7507 (time 1709787145.54 total 357.32 last 21.00) - open/close 7709 (time 1709787155.56 total 367.33 last 20.17) - open/close 7895 (time 1709787165.84 total 377.62 last 18.09) - open/close 8113 (time 1709787176.32 total 388.09 last 20.81) - open/close 8272 (time 1709787186.57 total 398.35 last 15.50) - open/close 8491 (time 1709787196.82 total 408.60 last 21.37) - open/close 8656 (time 1709787206.84 total 418.61 last 16.48) - open/close 8802 (time 1709787217.15 total 428.92 last 14.16) - open/close 8999 (time 1709787227.26 total 439.04 last 19.48) - open/close 9242 (time 1709787237.54 total 449.32 last 23.64) - open/close 9446 (time 1709787247.92 total 459.69 last 19.66) - open/close 9679 (time 1709787258.86 total 470.64 last 21.29) - open/close 9838 (time 1709787269.87 total 481.65 last 14.44) - open/close 10000 (time 1709787274.24 total 486.01 last 37.08) - open/close 10149 (time 1709787285.21 total 496.98 last 13.59) - open/close 10344 (time 1709787295.22 total 506.99 last 19.47) - open/close 10465 (time 1709787305.81 total 517.59 last 11.42) - open/close 10610 (time 1709787315.81 total 527.59 last 14.50) - open/close 11072 (time 1709787325.84 total 537.61 last 46.08) - open/close 11620 (time 1709787335.84 total 547.62 last 54.79) - open/close 12100 (time 1709787345.85 total 557.62 last 47.97) - open/close 12586 (time 1709787355.86 total 567.63 last 48.53) - open/close 13070 (time 1709787365.87 total 577.64 last 48.36) - open/close 13578 (time 1709787375.88 total 587.66 last 50.73) - open/close 14024 (time 1709787385.89 total 597.66 last 44.57) - open/close 14439 (time 1709787395.90 total 607.68 last 41.44) - open/close 14837 (time 1709787405.91 total 617.68 last 39.78) - open/close 15009 (time 1709787421.02 total 632.79 last 11.38) - open/close 15431 (time 1709787431.04 total 642.81 last 42.10) - open/close 15874 (time 1709787441.05 total 652.83 last 44.25) - open/close 15878 (time 1709787452.19 total 663.97 last 0.36) - open/close 16256 (time 1709787462.23 total 674.00 last 37.67) - open/close 16629 (time 1709787472.24 total 684.01 last 37.26) - open/close 16995 (time 1709787482.25 total 694.02 last 36.56) - open/close 17217 (time 1709787498.76 total 710.53 last 13.45) - open/close 17621 (time 1709787508.78 total 720.55 last 40.32) - open/close 17980 (time 1709787518.79 total 730.57 last 35.85) - open/close 18398 (time 1709787528.80 total 740.58 last 41.75) - open/close 18796 (time 1709787538.81 total 750.58 last 39.78) - open/close 19129 (time 1709787548.81 total 760.59 last 33.28) - open/close 19444 (time 1709787558.82 total 770.60 last 31.48) - open/close 19753 (time 1709787568.84 total 780.62 last 30.83) - open/close 20000 (time 1709787575.86 total 787.64 last 35.19) - open/close 20346 (time 1709787585.87 total 797.65 last 34.58) - open/close 20710 (time 1709787595.88 total 807.66 last 36.36) - open/close 21075 (time 1709787605.90 total 817.68 last 36.43) - open/close 21455 (time 1709787615.91 total 827.69 last 37.97) - open/close 21809 (time 1709787625.92 total 837.69 last 35.38) - open/close 22112 (time 1709787635.92 total 847.70 last 30.28) - open/close 22454 (time 1709787645.94 total 857.72 last 34.13) - open/close 22787 (time 1709787655.96 total 867.74 last 33.25) - open/close 23085 (time 1709787665.98 total 877.76 last 29.73) - open/close 23383 (time 1709787676.01 total 887.78 last 29.73) - open/close 23715 (time 1709787686.01 total 897.78 last 33.19) - open/close 24036 (time 1709787696.02 total 907.80 last 32.06) - open/close 24399 (time 1709787706.05 total 917.82 last 36.21) - open/close 24719 (time 1709787716.06 total 927.83 last 31.97) - open/close 25087 (time 1709787726.07 total 937.84 last 36.76) - open/close 25444 (time 1709787736.08 total 947.85 last 35.67) - open/close 25754 (time 1709787746.08 total 957.86 last 30.98) - open/close 26055 (time 1709787756.09 total 967.87 last 30.07) - open/close 26359 (time 1709787766.10 total 977.88 last 30.36) - open/close 26664 (time 1709787776.12 total 987.89 last 30.46) - open/close 26957 (time 1709787786.16 total 997.93 last 29.18) - open/close 27256 (time 1709787796.20 total 1007.97 last 29.79) - open/close 27574 (time 1709787806.20 total 1017.98 last 31.77) - open/close 27905 (time 1709787816.21 total 1027.98 last 33.08) - open/close 28203 (time 1709787826.22 total 1038.00 last 29.76) - open/close 28475 (time 1709787836.24 total 1048.01 last 27.16) - open/close 28738 (time 1709787846.28 total 1058.06 last 26.19) - open/close 29016 (time 1709787856.35 total 1068.13 last 27.61) - open/close 29298 (time 1709787866.39 total 1078.16 last 28.10) - open/close 29580 (time 1709787876.41 total 1088.18 last 28.14) - open/close 29839 (time 1709787886.42 total 1098.19 last 25.88) - open/close 30000 (time 1709787892.37 total 1104.14 last 27.05) - open/close 30267 (time 1709787902.38 total 1114.16 last 26.65) - open/close 30534 (time 1709787912.41 total 1124.18 last 26.64) - open/close 30781 (time 1709787922.46 total 1134.23 last 24.58) - open/close 31043 (time 1709787932.47 total 1144.25 last 26.16) - open/close 31299 (time 1709787942.52 total 1154.30 last 25.48) - open/close 31492 (time 1709787961.32 total 1173.10 last 10.26) - open/close 31715 (time 1709787980.39 total 1192.16 last 11.70) - open/close 32039 (time 1709787990.42 total 1202.19 last 32.31) - open/close 32345 (time 1709788000.42 total 1212.20 last 30.58) - open/close 32641 (time 1709788010.43 total 1222.20 last 29.59) - open/close 32945 (time 1709788020.45 total 1232.23 last 30.32) - open/close 33210 (time 1709788030.45 total 1242.23 last 26.50) - open/close 33495 (time 1709788040.48 total 1252.26 last 28.42) - open/close 33800 (time 1709788050.49 total 1262.26 last 30.48) - open/close 34104 (time 1709788060.50 total 1272.27 last 30.38) - open/close 34396 (time 1709788070.52 total 1282.30 last 29.12) - open/close 34712 (time 1709788080.54 total 1292.32 last 31.54) - open/close 35017 (time 1709788090.57 total 1302.35 last 30.41) - open/close 35312 (time 1709788100.60 total 1312.37 last 29.42) - open/close 35593 (time 1709788110.63 total 1322.41 last 28.00) - open/close 35885 (time 1709788120.66 total 1332.44 last 29.11) - open/close 36143 (time 1709788130.68 total 1342.46 last 25.76) - open/close 36425 (time 1709788140.69 total 1352.47 last 28.16) - open/close 36691 (time 1709788150.69 total 1362.47 last 26.60) - open/close 36951 (time 1709788160.73 total 1372.51 last 25.91) - open/close 37216 (time 1709788170.76 total 1382.53 last 26.43) - open/close 37484 (time 1709788180.76 total 1392.54 last 26.78) - open/close 37758 (time 1709788190.78 total 1402.56 last 27.35) - open/close 38028 (time 1709788200.81 total 1412.59 last 26.92) - open/close 38318 (time 1709788210.83 total 1422.60 last 28.95) - open/close 38596 (time 1709788220.85 total 1432.62 last 27.75) - open/close 38877 (time 1709788230.86 total 1442.63 last 28.08) - open/close 39147 (time 1709788240.88 total 1452.65 last 26.93) - open/close 39392 (time 1709788250.88 total 1462.66 last 24.50) - open/close 39623 (time 1709788260.91 total 1472.68 last 23.04) - open/close 39877 (time 1709788270.93 total 1482.70 last 25.34) - open/close 40000 (time 1709788275.72 total 1487.49 last 25.68) - open/close 40265 (time 1709788285.75 total 1497.53 last 26.41) - open/close 40526 (time 1709788295.77 total 1507.55 last 26.05) - open/close 40788 (time 1709788305.78 total 1517.56 last 26.17) - open/close 41038 (time 1709788315.78 total 1527.56 last 24.99) - open/close 41295 (time 1709788325.80 total 1537.57 last 25.67) - open/close 41552 (time 1709788335.82 total 1547.60 last 25.63) - open/close 41795 (time 1709788345.86 total 1557.64 last 24.20) - open/close 42043 (time 1709788355.88 total 1567.65 last 24.76) - open/close 42300 (time 1709788365.91 total 1577.68 last 25.62) - open/close 42550 (time 1709788375.96 total 1587.73 last 24.88) - open/close 42795 (time 1709788385.99 total 1597.76 last 24.43) - open/close 43039 (time 1709788395.99 total 1607.77 last 24.39) - open/close 43283 (time 1709788406.02 total 1617.79 last 24.34) - open/close 43533 (time 1709788416.06 total 1627.83 last 24.90) - open/close 43766 (time 1709788426.09 total 1637.87 last 23.21) - open/close 44013 (time 1709788436.12 total 1647.90 last 24.63) - open/close 44244 (time 1709788446.13 total 1657.91 last 23.08) - open/close 44486 (time 1709788456.16 total 1667.94 last 24.13) - open/close 44723 (time 1709788466.18 total 1677.95 last 23.67) - open/close 44971 (time 1709788476.18 total 1687.96 last 24.79) - open/close 45209 (time 1709788486.20 total 1697.97 last 23.76) - open/close 45458 (time 1709788496.21 total 1707.99 last 24.87) - open/close 45697 (time 1709788506.22 total 1717.99 last 23.89) - open/close 45938 (time 1709788516.24 total 1728.02 last 24.03) - open/close 46174 (time 1709788526.25 total 1738.03 last 23.58) - open/close 46414 (time 1709788536.27 total 1748.04 last 23.96) - open/close 46655 (time 1709788546.29 total 1758.07 last 24.05) - open/close 46893 (time 1709788556.31 total 1768.09 last 23.74) - open/close 47130 (time 1709788566.34 total 1778.11 last 23.65) - open/close 47364 (time 1709788576.35 total 1788.13 last 23.36) - open/close 47588 (time 1709788586.39 total 1798.17 last 22.31) - open/close 47823 (time 1709788596.43 total 1808.20 last 23.42) - open/close 48049 (time 1709788606.43 total 1818.21 last 22.59) - open/close 48284 (time 1709788616.46 total 1828.24 last 23.43) - open/close 48522 (time 1709788626.48 total 1838.25 last 23.76) - open/close 48751 (time 1709788636.49 total 1848.26 last 22.88) - open/close 48964 (time 1709788646.49 total 1858.27 last 21.29) - open/close 49186 (time 1709788656.51 total 1868.28 last 22.16) - open/close 49406 (time 1709788666.55 total 1878.33 last 21.90) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No space left on device total: 49600 open/close in 1887.35 seconds: 26.28 ops/second - unlinked 0 (time 1709788677 ; total 0 ; last 0) - unlinked 10000 (time 1709789205 ; total 528 ; last 528) - unlinked 20000 (time 1709789667 ; total 990 ; last 462) - unlinked 30000 (time 1709790012 ; total 1335 ; last 345) - unlinked 40000 (time 1709790207 ; total 1530 ; last 195) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No such file or directory total: 49600 unlinks in 1594 seconds: 31.116688 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg133-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg133-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg133-server stop mds service on oleg133-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg133-server stop mds service on oleg133-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg133-server unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg133-client: oleg133-client: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-client: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 101a (3543s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 00:44:56 (1709790296) start mds service on oleg133-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing load_modules_local oleg133-server: Loading modules from /home/green/git/lustre-release/lustre oleg133-server: detected 4 online CPUs by sysfs oleg133-server: Force libcfs to create 2 CPU partitions oleg133-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg133-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg133-server: oleg133-server.virtnet: executing set_default_debug -1 all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg133-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg133-server: oleg133-server.virtnet: executing set_default_debug -1 all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg133-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg133-server: oleg133-server.virtnet: executing set_default_debug -1 all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Started lustre-OST0000 oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg133-client.virtnet: -o user_xattr,flock oleg133-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg133-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg133-server