== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 22:59:04 (1788836344) Stopping clients: oleg629-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg629-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg629-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg629-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg629-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg629-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg629-server oleg629-server: oleg629-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg629-server' oleg629-server: oleg629-server.virtnet: executing load_modules_local oleg629-server: kmemleak disabled oleg629-server: Loading modules from /home/green/git/lustre-release/lustre oleg629-server: detected 4 online CPUs by sysfs oleg629-server: MODOPTS_LIBCFS= oleg629-server: Force libcfs to create 2 CPU partitions oleg629-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg629-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg629-server: oleg629-server.virtnet: executing set_default_debug -1 all pdsh@oleg629-client: oleg629-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg629-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg629-server: oleg629-server.virtnet: executing set_default_debug -1 all pdsh@oleg629-client: oleg629-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg629-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg629-server: oleg629-server.virtnet: executing set_default_debug -1 all pdsh@oleg629-client: oleg629-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg629-client.virtnet: -o user_xattr,flock 192.168.206.129@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 925 (time 1788836535.75 total 10.01 last 92.43) - open/close 1794 (time 1788836545.75 total 20.01 last 86.88) - open/close 2708 (time 1788836555.75 total 30.01 last 91.37) - open/close 3616 (time 1788836565.76 total 40.02 last 90.76) total: 4500 open/close in 49.93 seconds: 90.13 ops/second - /unlink 1741 (time 1788836589.62 total 10.00 last 174.05) - /unlink 3355 (time 1788836599.63 total 20.01 last 161.36) total: 4500 /unlink in 26.86 seconds: 167.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 862 (time 1788836620.11 total 10.00 last 86.16) - open/close 1725 (time 1788836630.11 total 20.01 last 86.29) - open/close 2595 (time 1788836640.11 total 30.01 last 87.00) - open/close 3489 (time 1788836650.12 total 40.01 last 89.34) - open/close 4413 (time 1788836660.12 total 50.01 last 92.39) total: 4500 open/close in 50.81 seconds: 88.57 ops/second - /unlink 1646 (time 1788836674.61 total 10.00 last 164.59) - /unlink 3340 (time 1788836684.61 total 20.00 last 169.34) total: 4500 /unlink in 27.23 seconds: 165.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 883 (time 1788836705.77 total 10.01 last 88.22) - open/close 1814 (time 1788836715.77 total 20.01 last 93.08) - open/close 2724 (time 1788836725.78 total 30.02 last 90.95) - open/close 3630 (time 1788836735.78 total 40.02 last 90.53) total: 4500 open/close in 49.99 seconds: 90.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1637 (time 1788836759.40 total 10.00 last 163.64) - /unlink 3290 (time 1788836769.41 total 20.01 last 165.20) total: 4500 /unlink in 27.05 seconds: 166.37 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 818 (time 1788836790.76 total 10.02 last 81.60) - open/close 1713 (time 1788836800.77 total 20.03 last 89.46) - open/close 2581 (time 1788836810.77 total 30.04 last 86.75) - open/close 3452 (time 1788836820.78 total 40.04 last 87.06) - open/close 4297 (time 1788836830.78 total 50.04 last 84.49) total: 4500 open/close in 52.38 seconds: 85.92 ops/second - /unlink 1577 (time 1788836847.59 total 10.00 last 157.63) - /unlink 3081 (time 1788836857.60 total 20.01 last 150.35) total: 4500 /unlink in 28.43 seconds: 158.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 867 (time 1788836880.42 total 10.00 last 86.69) - open/close 1695 (time 1788836890.42 total 20.01 last 82.75) - open/close 2551 (time 1788836900.43 total 30.01 last 85.56) - open/close 3501 (time 1788836910.43 total 40.02 last 94.97) - open/close 4441 (time 1788836920.44 total 50.02 last 93.94) total: 4500 open/close in 50.70 seconds: 88.76 ops/second - /unlink 1831 (time 1788836935.42 total 10.00 last 183.03) - /unlink 3565 (time 1788836945.42 total 20.01 last 173.36) total: 4500 /unlink in 25.50 seconds: 176.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 912 (time 1788836965.02 total 10.01 last 91.13) - open/close 1810 (time 1788836975.02 total 20.01 last 89.77) - open/close 2645 (time 1788836985.03 total 30.02 last 83.46) - open/close 3455 (time 1788836995.03 total 40.02 last 80.99) - open/close 4315 (time 1788837005.04 total 50.03 last 85.90) total: 4500 open/close in 52.11 seconds: 86.36 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1671 (time 1788837021.36 total 10.01 last 167.00) - /unlink 3217 (time 1788837031.37 total 20.01 last 154.47) total: 4500 /unlink in 27.83 seconds: 161.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 831 (time 1788837053.88 total 10.00 last 83.09) - open/close 1728 (time 1788837063.89 total 20.02 last 89.54) - open/close 2618 (time 1788837073.90 total 30.02 last 88.98) - open/close 3405 (time 1788837083.90 total 40.02 last 78.67) - open/close 4252 (time 1788837093.91 total 50.03 last 84.64) total: 4500 open/close in 52.78 seconds: 85.27 ops/second - /unlink 1846 (time 1788837110.53 total 10.00 last 184.53) - /unlink 3609 (time 1788837120.53 total 20.01 last 176.22) total: 4500 /unlink in 25.72 seconds: 174.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 922 (time 1788837139.62 total 10.01 last 92.11) - open/close 1911 (time 1788837149.62 total 20.01 last 98.87) - open/close 2783 (time 1788837159.62 total 30.02 last 87.18) - open/close 3622 (time 1788837169.64 total 40.03 last 83.77) total: 4500 open/close in 49.30 seconds: 91.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1712 (time 1788837193.53 total 10.00 last 171.16) - /unlink 3289 (time 1788837203.53 total 20.00 last 157.66) total: 4500 /unlink in 27.60 seconds: 163.07 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 874 (time 1788837225.49 total 10.00 last 87.39) - open/close 1737 (time 1788837235.50 total 20.01 last 86.19) - open/close 2670 (time 1788837245.50 total 30.02 last 93.25) - open/close 3572 (time 1788837255.52 total 40.03 last 90.06) - open/close 4282 (time 1788837265.52 total 50.03 last 71.00) total: 4500 open/close in 52.80 seconds: 85.22 ops/second - /unlink 1704 (time 1788837281.91 total 10.01 last 170.20) - /unlink 2893 (time 1788837291.91 total 20.01 last 118.89) - /unlink 4301 (time 1788837301.91 total 30.02 last 140.75) total: 4500 /unlink in 31.71 seconds: 141.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 778 (time 1788837319.06 total 10.00 last 77.77) - open/close 1551 (time 1788837329.08 total 20.03 last 77.12) - open/close 2436 (time 1788837339.08 total 30.03 last 88.49) - open/close 3412 (time 1788837349.09 total 40.03 last 97.54) - open/close 4281 (time 1788837359.09 total 50.04 last 86.88) total: 4500 open/close in 52.50 seconds: 85.71 ops/second - /unlink 1562 (time 1788837375.92 total 10.00 last 156.18) - /unlink 3023 (time 1788837385.93 total 20.00 last 146.06) total: 4500 /unlink in 29.38 seconds: 153.15 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 886 (time 1788837410.11 total 10.02 last 88.43) - open/close 1777 (time 1788837420.11 total 20.02 last 89.09) - open/close 2642 (time 1788837430.11 total 30.02 last 86.49) - open/close 3518 (time 1788837440.12 total 40.03 last 87.53) - open/close 4406 (time 1788837450.13 total 50.04 last 88.75) total: 4500 open/close in 51.22 seconds: 87.85 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1461 (time 1788837465.89 total 10.00 last 146.07) - /unlink 2906 (time 1788837475.90 total 20.01 last 144.38) - /unlink 4336 (time 1788837485.91 total 30.02 last 142.86) total: 4500 /unlink in 31.37 seconds: 143.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 853 (time 1788837502.20 total 10.00 last 85.30) - open/close 1592 (time 1788837512.20 total 20.00 last 73.90) - open/close 2435 (time 1788837522.21 total 30.01 last 84.24) - open/close 3196 (time 1788837532.22 total 40.02 last 76.03) - open/close 4018 (time 1788837542.22 total 50.02 last 82.16) total: 4500 open/close in 55.34 seconds: 81.32 ops/second - /unlink 1660 (time 1788837561.47 total 10.00 last 165.95) - /unlink 3341 (time 1788837571.47 total 20.00 last 168.08) total: 4500 /unlink in 26.99 seconds: 166.76 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 893 (time 1788837592.77 total 10.01 last 89.22) - open/close 1764 (time 1788837602.78 total 20.02 last 87.04) - open/close 2709 (time 1788837612.78 total 30.02 last 94.44) - open/close 3608 (time 1788837622.79 total 40.03 last 89.86) - open/close 4480 (time 1788837632.81 total 50.04 last 87.04) total: 4500 open/close in 50.29 seconds: 89.48 ops/second - /unlink 1736 (time 1788837646.95 total 10.00 last 173.55) - /unlink 3297 (time 1788837656.95 total 20.01 last 156.04) total: 4500 /unlink in 27.57 seconds: 163.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 840 (time 1788837678.80 total 10.00 last 83.98) - open/close 1528 (time 1788837688.81 total 20.02 last 68.71) - open/close 2416 (time 1788837698.83 total 30.03 last 88.66) - open/close 3323 (time 1788837708.84 total 40.04 last 90.58) - open/close 4161 (time 1788837718.85 total 50.05 last 83.73) total: 4500 open/close in 53.99 seconds: 83.35 ops/second - /unlink 1628 (time 1788837736.83 total 10.00 last 162.77) - /unlink 3111 (time 1788837746.83 total 20.00 last 148.27) total: 4500 /unlink in 28.56 seconds: 157.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 830 (time 1788837770.62 total 10.02 last 82.85) - open/close 1726 (time 1788837780.63 total 20.02 last 89.56) - open/close 2655 (time 1788837790.63 total 30.02 last 92.90) - open/close 3495 (time 1788837800.64 total 40.03 last 83.93) - open/close 4332 (time 1788837810.64 total 50.04 last 83.66) total: 4500 open/close in 51.90 seconds: 86.70 ops/second - /unlink 1744 (time 1788837826.58 total 10.00 last 174.38) - /unlink 3407 (time 1788837836.58 total 20.00 last 166.27) total: 4500 /unlink in 26.71 seconds: 168.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 897 (time 1788837857.56 total 10.01 last 89.62) - open/close 1770 (time 1788837867.57 total 20.02 last 87.22) - open/close 2602 (time 1788837877.58 total 30.03 last 83.14) - open/close 3462 (time 1788837887.58 total 40.03 last 85.99) - open/close 4344 (time 1788837897.58 total 50.03 last 88.17) total: 4500 open/close in 51.70 seconds: 87.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1676 (time 1788837913.64 total 10.00 last 167.59) - /unlink 3400 (time 1788837923.64 total 20.01 last 172.31) total: 4500 /unlink in 26.98 seconds: 166.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 860 (time 1788837944.40 total 10.00 last 85.98) - open/close 1643 (time 1788837954.40 total 20.00 last 78.29) - open/close 2476 (time 1788837964.43 total 30.03 last 83.09) - open/close 3380 (time 1788837974.43 total 40.03 last 90.38) - open/close 4185 (time 1788837984.43 total 50.04 last 80.47) total: 4500 open/close in 53.73 seconds: 83.75 ops/second - /unlink 1796 (time 1788838002.82 total 10.00 last 179.59) - /unlink 3483 (time 1788838012.82 total 20.00 last 168.68) total: 4500 /unlink in 26.53 seconds: 169.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 807 (time 1788838033.23 total 10.00 last 80.68) - open/close 1573 (time 1788838043.23 total 20.00 last 76.59) - open/close 2314 (time 1788838053.23 total 30.00 last 74.10) - open/close 3160 (time 1788838063.24 total 40.01 last 84.51) - open/close 3998 (time 1788838073.24 total 50.02 last 83.78) total: 4500 open/close in 55.67 seconds: 80.84 ops/second - /unlink 1725 (time 1788838093.09 total 10.00 last 172.49) - /unlink 3323 (time 1788838103.09 total 20.00 last 159.77) total: 4500 /unlink in 27.48 seconds: 163.78 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 95237 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg629-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg629-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg629-server stop mds service on oleg629-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg629-server stop mds service on oleg629-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg629-server unloading modules via unload_modules_local on: 'oleg629-server' oleg629-server: oleg629-server.virtnet: executing unload_modules_local oleg629-server: kmemleak disabled modules unloaded.