== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 05:36:20 (1744536980) start mds service on oleg452-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg452-server' oleg452-server: oleg452-server.virtnet: executing load_modules_local oleg452-server: Loading modules from /home/green/git/lustre-release/lustre oleg452-server: detected 4 online CPUs by sysfs oleg452-server: MODOPTS_LIBCFS= oleg452-server: Force libcfs to create 2 CPU partitions oleg452-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg452-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg452-server: oleg452-server.virtnet: executing set_default_debug -1 all pdsh@oleg452-client: oleg452-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg452-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg452-server: oleg452-server.virtnet: executing set_default_debug -1 all pdsh@oleg452-client: oleg452-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg452-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg452-server: oleg452-server.virtnet: executing set_default_debug -1 all pdsh@oleg452-client: oleg452-server: ssh exited with exit code 1 Started lustre-OST0000 oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg452-client.virtnet: -o user_xattr,flock 192.168.204.152@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 654 (time 1744537022.47 total 12.47 last 52.45) - open/close 1295 (time 1744537032.72 total 22.72 last 62.55) - open/close 1987 (time 1744537043.12 total 33.12 last 66.54) - open/close 2782 (time 1744537053.82 total 43.82 last 74.30) - open/close 3551 (time 1744537064.37 total 54.37 last 72.90) - open/close 4158 (time 1744537074.54 total 64.54 last 59.65) - open/close 4846 (time 1744537084.99 total 74.99 last 65.82) - open/close 5595 (time 1744537095.44 total 85.44 last 71.70) - open/close 6111 (time 1744537106.51 total 96.51 last 46.63) - open/close 6919 (time 1744537117.05 total 107.05 last 76.66) - open/close 7663 (time 1744537127.76 total 117.76 last 69.42) - open/close 8567 (time 1744537138.54 total 128.54 last 83.92) - open/close 9307 (time 1744537149.00 total 139.00 last 70.74) - open/close 9817 (time 1744537159.17 total 149.17 last 50.16) - open/close 10000 (time 1744537161.53 total 151.53 last 77.29) - open/close 10759 (time 1744537172.06 total 162.06 last 72.07) - open/close 11503 (time 1744537182.44 total 172.44 last 71.74) - open/close 12267 (time 1744537192.92 total 182.92 last 72.84) - open/close 12852 (time 1744537203.09 total 193.09 last 57.53) - open/close 13498 (time 1744537213.47 total 203.47 last 62.27) - open/close 14020 (time 1744537223.56 total 213.56 last 51.71) - open/close 14632 (time 1744537233.75 total 223.75 last 60.07) - open/close 15373 (time 1744537244.22 total 234.22 last 70.76) - open/close 18644 (time 1744537254.22 total 244.22 last 327.10) - open/close 20000 (time 1744537256.94 total 246.95 last 497.73) - open/close 24955 (time 1744537266.95 total 256.95 last 495.43) - open/close 28788 (time 1744537276.95 total 266.95 last 383.14) - open/close 30000 (time 1744537281.58 total 271.58 last 261.89) - open/close 32456 (time 1744537291.58 total 281.58 last 245.50) - open/close 35134 (time 1744537301.59 total 291.59 last 267.72) - open/close 38887 (time 1744537311.59 total 301.59 last 375.27) - open/close 40000 (time 1744537313.73 total 303.73 last 519.52) - open/close 44470 (time 1744537323.73 total 313.73 last 446.97) - open/close 48111 (time 1744537333.73 total 323.73 last 364.02) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 327.28 seconds: 151.65 ops/second - unlinked 0 (time 1744537338 ; total 0 ; last 0) - unlinked 10000 (time 1744537348 ; total 10 ; last 10) - unlinked 20000 (time 1744537359 ; total 21 ; last 11) - unlinked 30000 (time 1744537370 ; total 32 ; last 11) - unlinked 40000 (time 1744537381 ; total 43 ; last 11) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 53 seconds: 936.452820 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg452-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg452-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg452-server stop mds service on oleg452-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg452-server stop mds service on oleg452-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg452-server unloading modules via unload_modules_local on: 'oleg452-server' oleg452-server: oleg452-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg452-client: oleg452-client: ssh exited with exit code 2 pdsh@oleg452-client: oleg452-server: ssh exited with exit code 2 pdsh@oleg452-client: oleg452-client: ssh exited with exit code 2 pdsh@oleg452-client: oleg452-server: ssh exited with exit code 2