== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 00:05:54 (1721880354) start mds service on oleg220-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: detected 4 online CPUs by sysfs oleg220-server: Force libcfs to create 2 CPU partitions oleg220-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg220-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg220-server: oleg220-server.virtnet: executing set_default_debug -1 all pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg220-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg220-server: oleg220-server.virtnet: executing set_default_debug -1 all pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg220-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg220-server: oleg220-server.virtnet: executing set_default_debug -1 all pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1 Started lustre-OST0000 oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg220-client.virtnet: -o user_xattr,flock oleg220-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 989 (time 1721880394.49 total 10.92 last 90.55) - open/close 1921 (time 1721880405.25 total 21.68 last 86.65) - open/close 2760 (time 1721880415.80 total 32.22 last 79.57) - open/close 3634 (time 1721880426.39 total 42.82 last 82.49) - open/close 4410 (time 1721880436.82 total 53.25 last 74.38) - open/close 5126 (time 1721880447.09 total 63.52 last 69.74) - open/close 5925 (time 1721880457.53 total 73.96 last 76.54) - open/close 6689 (time 1721880467.93 total 84.36 last 73.45) - open/close 7536 (time 1721880478.46 total 94.88 last 80.46) - open/close 8274 (time 1721880488.80 total 105.23 last 71.34) - open/close 9234 (time 1721880499.67 total 116.10 last 88.32) - open/close 9963 (time 1721880510.05 total 126.48 last 70.22) - open/close 10768 (time 1721880520.52 total 136.95 last 76.91) - open/close 11354 (time 1721880530.70 total 147.13 last 57.55) - open/close 12023 (time 1721880541.05 total 157.47 last 64.68) - open/close 12964 (time 1721880551.80 total 168.23 last 87.49) - open/close 13744 (time 1721880562.30 total 178.72 last 74.34) - open/close 14525 (time 1721880572.79 total 189.21 last 74.45) - open/close 15254 (time 1721880583.14 total 199.57 last 70.39) - open/close 16076 (time 1721880593.67 total 210.10 last 78.04) - open/close 16941 (time 1721880604.32 total 220.74 last 81.28) - open/close 17839 (time 1721880615.02 total 231.44 last 83.92) - open/close 20000 (time 1721880622.94 total 239.37 last 272.65) - open/close 25419 (time 1721880632.95 total 249.37 last 541.84) - open/close 30000 (time 1721880641.27 total 257.69 last 550.47) - open/close 35610 (time 1721880651.27 total 267.69 last 560.93) - open/close 40000 (time 1721880659.23 total 275.66 last 551.14) - open/close 45450 (time 1721880669.23 total 285.66 last 544.99) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 293.35 seconds: 169.19 ops/second - unlinked 0 (time 1721880677 ; total 0 ; last 0) - unlinked 10000 (time 1721880688 ; total 11 ; last 11) - unlinked 20000 (time 1721880698 ; total 21 ; last 10) - unlinked 30000 (time 1721880708 ; total 31 ; last 10) - unlinked 40000 (time 1721880719 ; total 42 ; last 11) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 51 seconds: 973.176453 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg220-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg220-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg220-server stop mds service on oleg220-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg220-server stop mds service on oleg220-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg220-server unloading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg220-client: oleg220-client: ssh exited with exit code 2 pdsh@oleg220-client: oleg220-server: ssh exited with exit code 2 pdsh@oleg220-client: oleg220-client: ssh exited with exit code 2 pdsh@oleg220-client: oleg220-server: ssh exited with exit code 2