== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 13:57:07 (1733165827) start mds service on oleg106-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg106-server' oleg106-server: oleg106-server.virtnet: executing load_modules_local oleg106-server: Loading modules from /home/green/git/lustre-release/lustre oleg106-server: detected 4 online CPUs by sysfs oleg106-server: Force libcfs to create 2 CPU partitions oleg106-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg106-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg106-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg106-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Started lustre-OST0000 oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg106-client.virtnet: -o user_xattr,flock 192.168.201.106@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 304 (time 1733165866.77 total 10.75 last 28.29) - open/close 933 (time 1733165876.82 total 20.80 last 62.55) - open/close 1732 (time 1733165887.50 total 31.48 last 74.82) - open/close 2419 (time 1733165898.99 total 42.97 last 59.79) - open/close 3219 (time 1733165909.55 total 53.53 last 75.77) - open/close 3884 (time 1733165919.75 total 63.73 last 65.17) - open/close 4798 (time 1733165930.49 total 74.47 last 85.13) - open/close 5459 (time 1733165940.81 total 84.79 last 64.05) - open/close 6330 (time 1733165951.56 total 95.54 last 80.98) - open/close 7148 (time 1733165962.12 total 106.10 last 77.52) - open/close 7635 (time 1733165974.17 total 118.16 last 40.38) - open/close 8387 (time 1733165984.75 total 128.73 last 71.10) - open/close 9183 (time 1733165995.32 total 139.30 last 75.30) - open/close 10000 (time 1733166004.93 total 148.92 last 85.00) - open/close 10659 (time 1733166017.38 total 161.37 last 52.93) - open/close 11414 (time 1733166028.02 total 172.00 last 71.02) - open/close 12015 (time 1733166038.35 total 182.33 last 58.16) - open/close 12763 (time 1733166048.88 total 192.86 last 71.02) - open/close 13650 (time 1733166059.63 total 203.61 last 82.50) - open/close 14149 (time 1733166070.61 total 214.59 last 45.47) - open/close 14655 (time 1733166082.70 total 226.68 last 41.83) - open/close 15147 (time 1733166092.70 total 236.68 last 49.20) - open/close 17863 (time 1733166102.70 total 246.68 last 271.57) - open/close 20000 (time 1733166107.58 total 251.56 last 438.21) - open/close 25896 (time 1733166117.58 total 261.56 last 589.58) - open/close 30000 (time 1733166125.18 total 269.16 last 539.75) - open/close 34936 (time 1733166135.18 total 279.16 last 493.59) - open/close 40000 (time 1733166144.64 total 288.62 last 535.49) - open/close 45415 (time 1733166154.64 total 298.62 last 541.45) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 307.25 seconds: 161.54 ops/second - unlinked 0 (time 1733166164 ; total 0 ; last 0) - unlinked 10000 (time 1733166174 ; total 10 ; last 10) - unlinked 20000 (time 1733166184 ; total 20 ; last 10) - unlinked 30000 (time 1733166194 ; total 30 ; last 10) - unlinked 40000 (time 1733166203 ; total 39 ; last 9) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 47 seconds: 1056.000000 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg106-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg106-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg106-server stop mds service on oleg106-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg106-server stop mds service on oleg106-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg106-server unloading modules via unload_modules_local on: 'oleg106-server' oleg106-server: oleg106-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg106-client: oleg106-client: ssh exited with exit code 2 pdsh@oleg106-client: oleg106-server: ssh exited with exit code 2 pdsh@oleg106-client: oleg106-client: ssh exited with exit code 2 pdsh@oleg106-client: oleg106-server: ssh exited with exit code 2