== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 08:18:01 (1746793081) start mds service on oleg146-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg146-server' oleg146-server: oleg146-server.virtnet: executing load_modules_local oleg146-server: Loading modules from /home/green/git/lustre-release/lustre oleg146-server: detected 4 online CPUs by sysfs oleg146-server: MODOPTS_LIBCFS= oleg146-server: Force libcfs to create 2 CPU partitions oleg146-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg146-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg146-server: oleg146-server.virtnet: executing set_default_debug -1 all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg146-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg146-server: oleg146-server.virtnet: executing set_default_debug -1 all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg146-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg146-server: oleg146-server.virtnet: executing set_default_debug -1 all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Started lustre-OST0000 oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg146-client.virtnet: -o user_xattr,flock 192.168.201.146@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 215 (time 1746793138.11 total 10.80 last 19.91) - open/close 431 (time 1746793150.97 total 23.66 last 16.80) - open/close 513 (time 1746793161.32 total 34.01 last 7.92) - open/close 586 (time 1746793171.65 total 44.34 last 7.07) - open/close 739 (time 1746793182.31 total 55.00 last 14.35) - open/close 853 (time 1746793194.82 total 67.51 last 9.12) - open/close 1106 (time 1746793204.82 total 77.51 last 25.29) - open/close 1201 (time 1746793215.24 total 87.93 last 9.11) - open/close 1287 (time 1746793226.62 total 99.31 last 7.56) - open/close 1445 (time 1746793237.27 total 109.96 last 14.85) - open/close 1496 (time 1746793248.49 total 121.18 last 4.54) - open/close 1693 (time 1746793259.28 total 131.97 last 18.26) - open/close 1971 (time 1746793269.36 total 142.05 last 27.59) - open/close 2094 (time 1746793279.87 total 152.56 last 11.70) - open/close 2202 (time 1746793290.33 total 163.02 last 10.32) - open/close 2526 (time 1746793300.42 total 173.11 last 32.10) - open/close 2793 (time 1746793310.43 total 183.12 last 26.68) - open/close 3010 (time 1746793321.23 total 193.92 last 20.10) - open/close 3292 (time 1746793331.42 total 204.11 last 27.69) - open/close 3457 (time 1746793342.05 total 214.74 last 15.51) - open/close 3480 (time 1746793354.18 total 226.87 last 1.90) - open/close 3714 (time 1746793364.18 total 236.87 last 23.40) - open/close 3862 (time 1746793374.81 total 247.50 last 13.92) - open/close 4095 (time 1746793385.76 total 258.45 last 21.27) - open/close 4271 (time 1746793396.49 total 269.18 last 16.41) - open/close 4610 (time 1746793407.39 total 280.08 last 31.10) - open/close 5090 (time 1746793417.54 total 290.23 last 47.29) - open/close 5637 (time 1746793427.95 total 300.64 last 52.54) - open/close 5808 (time 1746793438.50 total 311.19 last 16.20) - open/close 5918 (time 1746793449.00 total 321.69 last 10.48) - open/close 7504 (time 1746793459.00 total 331.69 last 158.57) - open/close 10000 (time 1746793469.01 total 341.70 last 249.45) - open/close 13056 (time 1746793479.01 total 351.70 last 305.57) - open/close 16452 (time 1746793489.01 total 361.70 last 339.58) - open/close 19260 (time 1746793499.01 total 371.70 last 280.80) - open/close 22134 (time 1746793509.01 total 381.70 last 287.35) - open/close 24682 (time 1746793519.01 total 391.70 last 254.72) - open/close 26973 (time 1746793529.01 total 401.70 last 229.10) - open/close 29461 (time 1746793539.02 total 411.71 last 248.71) - open/close 30000 (time 1746793541.20 total 413.89 last 247.38) - open/close 32423 (time 1746793551.20 total 423.89 last 242.23) - open/close 34856 (time 1746793561.20 total 433.89 last 243.27) - open/close 37577 (time 1746793571.20 total 443.89 last 272.07) - open/close 40000 (time 1746793580.16 total 452.85 last 270.49) - open/close 42615 (time 1746793590.16 total 462.85 last 261.42) - open/close 45324 (time 1746793600.16 total 472.85 last 270.88) - open/close 47765 (time 1746793610.16 total 482.85 last 244.08) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 490.56 seconds: 101.17 ops/second - unlinked 0 (time 1746793619 ; total 0 ; last 0) - unlinked 10000 (time 1746793640 ; total 21 ; last 21) - unlinked 20000 (time 1746793663 ; total 44 ; last 23) - unlinked 30000 (time 1746793687 ; total 68 ; last 24) - unlinked 40000 (time 1746793708 ; total 89 ; last 21) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 110 seconds: 451.200012 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg146-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg146-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg146-server stop mds service on oleg146-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg146-server stop mds service on oleg146-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg146-server unloading modules via unload_modules_local on: 'oleg146-server' oleg146-server: oleg146-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg146-client: oleg146-client: ssh exited with exit code 2 pdsh@oleg146-client: oleg146-server: ssh exited with exit code 2 pdsh@oleg146-client: oleg146-client: ssh exited with exit code 2 pdsh@oleg146-client: oleg146-server: ssh exited with exit code 2