== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 13:49:18 (1758736158) start mds service on oleg255-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing load_modules_local oleg255-server: Loading modules from /home/green/git/lustre-release/lustre oleg255-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg255-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg255-server: detected 4 online CPUs by sysfs oleg255-server: MODOPTS_LIBCFS= oleg255-server: Force libcfs to create 2 CPU partitions oleg255-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg255-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg255-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg255-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-OST0000 oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg255-client.virtnet: -o user_xattr,flock 192.168.202.155@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 810 (time 1758736217.14 total 10.18 last 79.60) - open/close 1609 (time 1758736227.19 total 20.22 last 79.52) - open/close 2201 (time 1758736237.82 total 30.85 last 55.72) - open/close 2783 (time 1758736248.44 total 41.47 last 54.79) - open/close 3422 (time 1758736258.87 total 51.91 last 61.24) - open/close 4243 (time 1758736268.98 total 62.02 last 81.20) - open/close 4974 (time 1758736278.98 total 72.02 last 73.10) - open/close 5823 (time 1758736289.14 total 82.18 last 83.57) - open/close 6600 (time 1758736299.14 total 92.18 last 77.69) - open/close 7350 (time 1758736309.15 total 102.18 last 74.99) - open/close 8140 (time 1758736319.16 total 112.19 last 78.89) - open/close 9017 (time 1758736329.46 total 122.50 last 85.11) - open/close 9790 (time 1758736339.64 total 132.67 last 75.95) - open/close 10690 (time 1758736350.20 total 143.23 last 85.22) - open/close 11280 (time 1758736361.15 total 154.18 last 53.89) - open/close 11765 (time 1758736371.96 total 164.99 last 44.87) - open/close 12397 (time 1758736382.71 total 175.75 last 58.77) - open/close 13058 (time 1758736393.40 total 186.43 last 61.86) - open/close 13637 (time 1758736403.90 total 196.93 last 55.15) - open/close 14269 (time 1758736414.52 total 207.55 last 59.49) - open/close 14995 (time 1758736425.40 total 218.43 last 66.72) - open/close 15798 (time 1758736435.64 total 228.67 last 78.41) - open/close 16648 (time 1758736445.94 total 238.98 last 82.51) - open/close 17341 (time 1758736456.12 total 249.15 last 68.10) - open/close 18058 (time 1758736466.12 total 259.15 last 71.70) - open/close 18770 (time 1758736477.11 total 270.15 last 64.77) - open/close 19504 (time 1758736488.12 total 281.16 last 66.66) - open/close 20000 (time 1758736494.45 total 287.48 last 78.40) - open/close 20647 (time 1758736505.14 total 298.17 last 60.52) - open/close 21223 (time 1758736515.83 total 308.86 last 53.88) - open/close 22268 (time 1758736525.83 total 318.87 last 104.48) - open/close 23288 (time 1758736535.84 total 328.87 last 101.93) - open/close 24812 (time 1758736545.84 total 338.87 last 152.38) - open/close 28539 (time 1758736555.84 total 348.88 last 372.65) - open/close 30000 (time 1758736559.67 total 352.70 last 382.10) - open/close 33821 (time 1758736569.67 total 362.70 last 382.05) - open/close 37637 (time 1758736579.67 total 372.70 last 381.57) - open/close 40000 (time 1758736586.11 total 379.14 last 366.78) - open/close 41635 (time 1758736596.13 total 389.16 last 163.28) - open/close 42336 (time 1758736606.13 total 399.16 last 70.05) - open/close 43039 (time 1758736616.14 total 409.17 last 70.22) - open/close 43616 (time 1758736626.15 total 419.18 last 57.65) - open/close 44213 (time 1758736636.16 total 429.19 last 59.66) - open/close 44875 (time 1758736646.17 total 439.20 last 66.15) - open/close 45760 (time 1758736656.18 total 449.21 last 88.37) - open/close 46828 (time 1758736666.18 total 459.22 last 106.76) - open/close 47743 (time 1758736676.20 total 469.23 last 91.38) - open/close 48761 (time 1758736686.20 total 479.23 last 101.79) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 488.02 seconds: 101.70 ops/second - unlinked 0 (time 1758736697 ; total 0 ; last 0) - unlinked 10000 (time 1758736748 ; total 51 ; last 51) - unlinked 20000 (time 1758736785 ; total 88 ; last 37) - unlinked 30000 (time 1758736829 ; total 132 ; last 44) - unlinked 40000 (time 1758736879 ; total 182 ; last 50) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 216 seconds: 229.777771 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg255-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg255-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg255-server stop mds service on oleg255-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg255-server stop mds service on oleg255-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg255-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing unload_modules_local oleg255-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg255-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg255-client: oleg255-client: ssh exited with exit code 2 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 2 pdsh@oleg255-client: oleg255-client: ssh exited with exit code 2 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 2