== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 00:14:55 (1744172095) start mds service on oleg353-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing load_modules_local oleg353-server: Loading modules from /home/green/git/lustre-release/lustre oleg353-server: detected 4 online CPUs by sysfs oleg353-server: MODOPTS_LIBCFS= oleg353-server: Force libcfs to create 2 CPU partitions oleg353-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg353-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg353-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg353-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg353-server: oleg353-server.virtnet: executing set_default_debug -1 all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg353-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg353-server: oleg353-server.virtnet: executing set_default_debug -1 all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg353-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg353-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg353-server: oleg353-server.virtnet: executing set_default_debug -1 all pdsh@oleg353-client: oleg353-server: ssh exited with exit code 1 Started lustre-OST0000 oleg353-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg353-client.virtnet: -o user_xattr,flock 192.168.203.153@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 255 (time 1744172147.24 total 10.01 last 25.46) - open/close 664 (time 1744172157.69 total 20.47 last 39.13) - open/close 776 (time 1744172168.16 total 30.94 last 10.70) - open/close 969 (time 1744172178.96 total 41.74 last 17.86) - open/close 1207 (time 1744172194.96 total 57.74 last 14.88) - open/close 1704 (time 1744172205.10 total 67.88 last 49.03) - open/close 2149 (time 1744172215.10 total 77.88 last 44.49) - open/close 2273 (time 1744172225.56 total 88.34 last 11.85) - open/close 2721 (time 1744172235.80 total 98.58 last 43.73) - open/close 3491 (time 1744172246.38 total 109.16 last 72.78) - open/close 4085 (time 1744172256.80 total 119.58 last 57.04) - open/close 4949 (time 1744172267.46 total 130.24 last 81.05) - open/close 5508 (time 1744172278.54 total 141.32 last 50.42) - open/close 6361 (time 1744172289.22 total 152.00 last 79.87) - open/close 6982 (time 1744172299.66 total 162.44 last 59.47) - open/close 7213 (time 1744172311.60 total 174.38 last 19.35) - open/close 7379 (time 1744172322.23 total 185.01 last 15.62) - open/close 7692 (time 1744172332.42 total 195.20 last 30.70) - open/close 8307 (time 1744172343.00 total 205.78 last 58.13) - open/close 8531 (time 1744172353.82 total 216.60 last 20.71) - open/close 8670 (time 1744172364.35 total 227.13 last 13.20) - open/close 8926 (time 1744172374.41 total 237.18 last 25.45) - open/close 9279 (time 1744172384.96 total 247.74 last 33.45) - open/close 9572 (time 1744172395.12 total 257.90 last 28.83) - open/close 9803 (time 1744172405.13 total 267.90 last 23.09) - open/close 10000 (time 1744172411.78 total 274.56 last 29.60) - open/close 10571 (time 1744172421.99 total 284.77 last 55.91) - open/close 12421 (time 1744172432.00 total 294.77 last 184.98) - open/close 15401 (time 1744172442.00 total 304.78 last 297.95) - open/close 17863 (time 1744172452.00 total 314.78 last 246.19) - open/close 20000 (time 1744172461.06 total 323.84 last 235.76) - open/close 22358 (time 1744172471.06 total 333.84 last 235.73) - open/close 24602 (time 1744172481.07 total 343.85 last 224.32) - open/close 26889 (time 1744172491.07 total 353.85 last 228.69) - open/close 29147 (time 1744172501.07 total 363.85 last 225.74) - open/close 30000 (time 1744172504.65 total 367.43 last 238.35) - open/close 32476 (time 1744172514.65 total 377.43 last 247.54) - open/close 35191 (time 1744172524.65 total 387.43 last 271.47) - open/close 37635 (time 1744172534.66 total 397.43 last 244.33) - open/close 39968 (time 1744172544.66 total 407.44 last 233.22) - open/close 42538 (time 1744172554.66 total 417.44 last 256.97) - open/close 46682 (time 1744172564.66 total 427.44 last 414.29) - open/close 49320 (time 1744172574.66 total 437.44 last 263.77) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 438.62 seconds: 113.15 ops/second - unlinked 0 (time 1744172576 ; total 0 ; last 0) - unlinked 10000 (time 1744172589 ; total 13 ; last 13) - unlinked 20000 (time 1744172600 ; total 24 ; last 11) - unlinked 30000 (time 1744172613 ; total 37 ; last 13) - unlinked 40000 (time 1744172624 ; total 48 ; last 11) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 60 seconds: 827.200012 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg353-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg353-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg353-server stop mds service on oleg353-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg353-server stop mds service on oleg353-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg353-server unloading modules via unload_modules_local on: 'oleg353-server' oleg353-server: oleg353-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg353-client: oleg353-client: ssh exited with exit code 2 pdsh@oleg353-client: oleg353-server: ssh exited with exit code 2 pdsh@oleg353-client: oleg353-client: ssh exited with exit code 2 pdsh@oleg353-client: oleg353-server: ssh exited with exit code 2