== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 07:40:38 (1749123638) start mds service on oleg159-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg159-server' oleg159-server: oleg159-server.virtnet: executing load_modules_local oleg159-server: Loading modules from /home/green/git/lustre-release/lustre oleg159-server: detected 4 online CPUs by sysfs oleg159-server: MODOPTS_LIBCFS= oleg159-server: Force libcfs to create 2 CPU partitions oleg159-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg159-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg159-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg159-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Started lustre-OST0000 oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg159-client.virtnet: -o user_xattr,flock 192.168.201.159@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 482 (time 1749123676.48 total 10.00 last 48.20) - open/close 1427 (time 1749123687.23 total 20.75 last 87.90) - open/close 2322 (time 1749123697.92 total 31.44 last 83.69) - open/close 3375 (time 1749123707.92 total 41.44 last 105.30) - open/close 4226 (time 1749123718.53 total 52.05 last 80.22) - open/close 4843 (time 1749123728.77 total 62.29 last 60.28) - open/close 5579 (time 1749123739.23 total 72.76 last 70.31) - open/close 6270 (time 1749123749.58 total 83.11 last 66.76) - open/close 7047 (time 1749123760.07 total 93.59 last 74.10) - open/close 7861 (time 1749123770.62 total 104.15 last 77.12) - open/close 8740 (time 1749123781.29 total 114.81 last 82.41) - open/close 9663 (time 1749123792.07 total 125.60 last 85.59) - open/close 10000 (time 1749123795.71 total 129.23 last 92.68) - open/close 10811 (time 1749123806.29 total 139.82 last 76.64) - open/close 11692 (time 1749123817.00 total 150.52 last 82.26) - open/close 12513 (time 1749123827.60 total 161.12 last 77.50) - open/close 13269 (time 1749123838.06 total 171.59 last 72.21) - open/close 14049 (time 1749123848.55 total 182.08 last 74.36) - open/close 14794 (time 1749123859.03 total 192.55 last 71.10) - open/close 15584 (time 1749123869.57 total 203.09 last 74.95) - open/close 16381 (time 1749123880.12 total 213.64 last 75.59) - open/close 17134 (time 1749123890.61 total 224.13 last 71.76) - open/close 17920 (time 1749123901.15 total 234.68 last 74.54) - open/close 20000 (time 1749123907.08 total 240.61 last 350.83) - open/close 25328 (time 1749123917.09 total 250.61 last 532.67) - open/close 30000 (time 1749123925.97 total 259.49 last 526.08) - open/close 35238 (time 1749123935.97 total 269.49 last 523.78) - open/close 40000 (time 1749123945.28 total 278.80 last 511.23) - open/close 44933 (time 1749123955.28 total 288.81 last 493.27) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 298.24 seconds: 166.42 ops/second - unlinked 0 (time 1749123965 ; total 0 ; last 0) - unlinked 10000 (time 1749123976 ; total 11 ; last 11) - unlinked 20000 (time 1749123986 ; total 21 ; last 10) - unlinked 30000 (time 1749123997 ; total 32 ; last 11) - unlinked 40000 (time 1749124007 ; total 42 ; last 10) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 53 seconds: 936.452820 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg159-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg159-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg159-server stop mds service on oleg159-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg159-server stop mds service on oleg159-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg159-server unloading modules via unload_modules_local on: 'oleg159-server' oleg159-server: oleg159-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg159-client: oleg159-client: ssh exited with exit code 2 pdsh@oleg159-client: oleg159-server: ssh exited with exit code 2 pdsh@oleg159-client: oleg159-client: ssh exited with exit code 2 pdsh@oleg159-client: oleg159-server: ssh exited with exit code 2