== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 13:38:43 (1741628323) start mds service on oleg337-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg337-server' oleg337-server: oleg337-server.virtnet: executing load_modules_local oleg337-server: Loading modules from /home/green/git/lustre-release/lustre oleg337-server: detected 4 online CPUs by sysfs oleg337-server: MODOPTS_LIBCFS= oleg337-server: Force libcfs to create 2 CPU partitions oleg337-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg337-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg337-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg337-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-OST0000 oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg337-client.virtnet: -o user_xattr,flock 192.168.203.137@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 344 (time 1741628363.95 total 10.98 last 31.33) - open/close 637 (time 1741628375.72 total 22.75 last 24.89) - open/close 1086 (time 1741628386.07 total 33.10 last 43.37) - open/close 1317 (time 1741628396.07 total 43.10 last 23.10) - open/close 1642 (time 1741628406.83 total 53.86 last 30.21) - open/close 1923 (time 1741628417.21 total 64.25 last 27.06) - open/close 2255 (time 1741628427.79 total 74.82 last 31.38) - open/close 2597 (time 1741628438.41 total 85.44 last 32.22) - open/close 2974 (time 1741628449.19 total 96.22 last 34.97) - open/close 3199 (time 1741628459.22 total 106.25 last 22.43) - open/close 3576 (time 1741628470.15 total 117.18 last 34.49) - open/close 3774 (time 1741628480.93 total 127.96 last 18.36) - open/close 3918 (time 1741628491.48 total 138.52 last 13.65) - open/close 4169 (time 1741628502.43 total 149.46 last 22.93) - open/close 4429 (time 1741628513.40 total 160.44 last 23.69) - open/close 4725 (time 1741628523.93 total 170.96 last 28.13) - open/close 5084 (time 1741628534.78 total 181.81 last 33.08) - open/close 5395 (time 1741628545.14 total 192.17 last 30.03) - open/close 5677 (time 1741628555.56 total 202.60 last 27.05) - open/close 5923 (time 1741628565.57 total 212.60 last 24.60) - open/close 6206 (time 1741628576.17 total 223.20 last 26.70) - open/close 6486 (time 1741628586.37 total 233.41 last 27.43) - open/close 6727 (time 1741628599.27 total 246.31 last 18.68) - open/close 7344 (time 1741628609.28 total 256.31 last 61.67) - open/close 7573 (time 1741628620.25 total 267.28 last 20.88) - open/close 7933 (time 1741628630.94 total 277.97 last 33.68) - open/close 9640 (time 1741628640.94 total 287.97 last 170.65) - open/close 12490 (time 1741628650.94 total 297.97 last 284.97) - open/close 15875 (time 1741628660.94 total 307.98 last 338.37) - open/close 19332 (time 1741628670.95 total 317.98 last 345.68) - open/close 20000 (time 1741628673.45 total 320.48 last 266.66) - open/close 22957 (time 1741628683.45 total 330.48 last 295.67) - open/close 26149 (time 1741628693.45 total 340.48 last 319.14) - open/close 29283 (time 1741628703.46 total 350.49 last 313.28) - open/close 30000 (time 1741628706.82 total 353.85 last 213.38) - open/close 32893 (time 1741628716.82 total 363.85 last 289.29) - open/close 36107 (time 1741628726.82 total 373.85 last 321.39) - open/close 39319 (time 1741628736.82 total 383.85 last 321.11) - open/close 43390 (time 1741628746.82 total 393.85 last 407.07) - open/close 47009 (time 1741628756.82 total 403.85 last 361.86) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 411.10 seconds: 120.73 ops/second - unlinked 0 (time 1741628764 ; total 0 ; last 0) - unlinked 10000 (time 1741628779 ; total 15 ; last 15) - unlinked 20000 (time 1741628795 ; total 31 ; last 16) - unlinked 30000 (time 1741628809 ; total 45 ; last 14) - unlinked 40000 (time 1741628822 ; total 58 ; last 13) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 77 seconds: 644.571411 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg337-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg337-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg337-server stop mds service on oleg337-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg337-server stop mds service on oleg337-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg337-server unloading modules via unload_modules_local on: 'oleg337-server' oleg337-server: oleg337-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg337-client: oleg337-client: ssh exited with exit code 2 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 2 pdsh@oleg337-client: oleg337-client: ssh exited with exit code 2 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 2