== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 01:59:21 (1787119161) Stopping clients: oleg146-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg146-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg146-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg146-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg146-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg146-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg146-server oleg146-server: oleg146-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg146-server' oleg146-server: oleg146-server.virtnet: executing load_modules_local oleg146-server: Loading modules from /home/green/git/lustre-release/lustre oleg146-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg146-server: detected 4 online CPUs by sysfs oleg146-server: MODOPTS_LIBCFS= oleg146-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg146-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg146-server: oleg146-server.virtnet: executing set_default_debug -1 all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg146-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg146-server: oleg146-server.virtnet: executing set_default_debug -1 all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg146-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg146-server: oleg146-server.virtnet: executing set_default_debug -1 all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg146-client.virtnet: -o user_xattr,flock 192.168.201.146@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 804 (time 1787119377.70 total 10.01 last 80.36) - open/close 1595 (time 1787119387.71 total 20.01 last 79.04) - open/close 2394 (time 1787119397.71 total 30.02 last 79.88) - open/close 3220 (time 1787119407.72 total 40.02 last 82.54) - open/close 3955 (time 1787119417.72 total 50.03 last 73.46) total: 4500 open/close in 56.56 seconds: 79.56 ops/second - /unlink 1527 (time 1787119439.00 total 10.02 last 152.45) - /unlink 3042 (time 1787119449.00 total 20.02 last 151.48) - /unlink 4494 (time 1787119459.01 total 30.02 last 145.11) total: 4500 /unlink in 30.05 seconds: 149.76 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 853 (time 1787119474.16 total 10.00 last 85.29) - open/close 1686 (time 1787119484.17 total 20.01 last 83.23) - open/close 2505 (time 1787119494.17 total 30.01 last 81.86) - open/close 3372 (time 1787119504.17 total 40.02 last 86.69) - open/close 4187 (time 1787119514.18 total 50.02 last 81.45) total: 4500 open/close in 53.46 seconds: 84.17 ops/second - /unlink 1526 (time 1787119532.02 total 10.00 last 152.59) - /unlink 3016 (time 1787119542.02 total 20.00 last 148.97) - /unlink 4424 (time 1787119552.02 total 30.00 last 140.78) total: 4500 /unlink in 30.76 seconds: 146.29 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 706 (time 1787119567.98 total 10.00 last 70.58) - open/close 1509 (time 1787119577.99 total 20.01 last 80.26) - open/close 2152 (time 1787119588.01 total 30.03 last 64.16) - open/close 2921 (time 1787119598.03 total 40.05 last 76.72) - open/close 3734 (time 1787119608.06 total 50.08 last 81.06) total: 4500 open/close in 59.13 seconds: 76.11 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1502 (time 1787119631.20 total 10.00 last 150.18) - /unlink 3049 (time 1787119641.20 total 20.00 last 154.68) total: 4500 /unlink in 29.14 seconds: 154.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 833 (time 1787119665.54 total 10.01 last 83.21) - open/close 1519 (time 1787119675.54 total 20.01 last 68.57) - open/close 2519 (time 1787119685.55 total 30.02 last 99.94) - open/close 3443 (time 1787119695.55 total 40.02 last 92.39) - open/close 4314 (time 1787119705.57 total 50.04 last 86.96) total: 4500 open/close in 52.54 seconds: 85.64 ops/second - /unlink 1563 (time 1787119722.28 total 10.00 last 156.27) - /unlink 3092 (time 1787119732.29 total 20.01 last 152.82) total: 4500 /unlink in 29.00 seconds: 155.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 908 (time 1787119756.26 total 10.01 last 90.74) - open/close 1693 (time 1787119766.26 total 20.01 last 78.45) - open/close 2351 (time 1787119776.26 total 30.01 last 65.80) - open/close 3059 (time 1787119786.26 total 40.02 last 70.79) - open/close 3928 (time 1787119796.27 total 50.02 last 86.82) total: 4500 open/close in 55.88 seconds: 80.53 ops/second - /unlink 1506 (time 1787119816.06 total 10.00 last 150.56) - /unlink 3119 (time 1787119826.07 total 20.01 last 161.22) total: 4500 /unlink in 29.07 seconds: 154.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 875 (time 1787119850.12 total 10.01 last 87.44) - open/close 1856 (time 1787119860.12 total 20.01 last 98.07) - open/close 2827 (time 1787119870.12 total 30.01 last 97.07) - open/close 3728 (time 1787119880.13 total 40.02 last 90.04) total: 4500 open/close in 49.19 seconds: 91.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1705 (time 1787119903.70 total 10.01 last 170.41) - /unlink 3139 (time 1787119913.70 total 20.01 last 143.37) total: 4500 /unlink in 28.71 seconds: 156.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 918 (time 1787119937.70 total 10.01 last 91.68) - open/close 1761 (time 1787119947.70 total 20.01 last 84.29) - open/close 2740 (time 1787119957.70 total 30.02 last 97.86) - open/close 3428 (time 1787119967.70 total 40.02 last 68.77) - open/close 4345 (time 1787119977.71 total 50.03 last 91.63) total: 4500 open/close in 51.87 seconds: 86.76 ops/second - /unlink 1443 (time 1787119993.95 total 10.01 last 144.21) - /unlink 2596 (time 1787120003.95 total 20.01 last 115.28) - /unlink 4224 (time 1787120013.96 total 30.01 last 162.75) total: 4500 /unlink in 32.19 seconds: 139.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 777 (time 1787120031.26 total 10.00 last 77.68) - open/close 1550 (time 1787120041.27 total 20.01 last 77.22) - open/close 2366 (time 1787120051.28 total 30.02 last 81.51) - open/close 3029 (time 1787120061.30 total 40.04 last 66.21) - open/close 3871 (time 1787120071.30 total 50.04 last 84.16) total: 4500 open/close in 56.89 seconds: 79.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1423 (time 1787120092.84 total 10.00 last 142.26) - /unlink 2973 (time 1787120102.84 total 20.00 last 154.97) - /unlink 4362 (time 1787120112.84 total 30.01 last 138.85) total: 4500 /unlink in 31.12 seconds: 144.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 822 (time 1787120128.65 total 10.01 last 82.12) - open/close 1486 (time 1787120138.65 total 20.01 last 66.38) - open/close 2368 (time 1787120148.66 total 30.02 last 88.16) - open/close 3201 (time 1787120158.66 total 40.02 last 83.27) - open/close 4078 (time 1787120168.66 total 50.02 last 87.66) total: 4500 open/close in 54.87 seconds: 82.01 ops/second - /unlink 1791 (time 1787120187.08 total 10.00 last 179.03) - /unlink 3211 (time 1787120197.09 total 20.01 last 141.84) total: 4500 /unlink in 27.94 seconds: 161.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 809 (time 1787120219.40 total 10.00 last 80.89) - open/close 1693 (time 1787120229.40 total 20.00 last 88.39) - open/close 2433 (time 1787120239.40 total 30.01 last 73.97) - open/close 3196 (time 1787120249.42 total 40.02 last 76.22) - open/close 4041 (time 1787120259.42 total 50.02 last 84.45) total: 4500 open/close in 55.65 seconds: 80.86 ops/second - /unlink 1734 (time 1787120279.03 total 10.00 last 173.35) - /unlink 3158 (time 1787120289.03 total 20.01 last 142.36) total: 4500 /unlink in 28.68 seconds: 156.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 832 (time 1787120312.56 total 10.01 last 83.15) - open/close 1692 (time 1787120322.57 total 20.02 last 85.90) - open/close 2640 (time 1787120332.57 total 30.02 last 94.79) - open/close 3442 (time 1787120342.58 total 40.02 last 80.15) - open/close 4230 (time 1787120352.59 total 50.04 last 78.70) total: 4500 open/close in 52.75 seconds: 85.31 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1509 (time 1787120369.57 total 10.00 last 150.85) - /unlink 2937 (time 1787120379.57 total 20.01 last 142.77) total: 4500 /unlink in 29.93 seconds: 150.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 916 (time 1787120403.68 total 10.01 last 91.47) - open/close 1713 (time 1787120413.68 total 20.02 last 79.67) - open/close 2552 (time 1787120423.69 total 30.02 last 83.86) - open/close 3394 (time 1787120433.69 total 40.03 last 84.15) - open/close 4136 (time 1787120443.70 total 50.03 last 74.15) total: 4500 open/close in 54.96 seconds: 81.88 ops/second - /unlink 1419 (time 1787120462.95 total 10.01 last 141.78) - /unlink 2772 (time 1787120472.96 total 20.01 last 135.26) - /unlink 4148 (time 1787120482.96 total 30.01 last 137.58) total: 4500 /unlink in 32.20 seconds: 139.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 821 (time 1787120499.26 total 10.01 last 82.06) - open/close 1649 (time 1787120509.26 total 20.01 last 82.77) - open/close 2487 (time 1787120519.26 total 30.01 last 83.79) - open/close 3398 (time 1787120529.26 total 40.01 last 91.10) - open/close 4261 (time 1787120539.26 total 50.01 last 86.29) total: 4500 open/close in 52.78 seconds: 85.25 ops/second - /unlink 1388 (time 1787120555.82 total 10.00 last 138.77) - /unlink 2856 (time 1787120565.83 total 20.01 last 146.70) - /unlink 4384 (time 1787120575.83 total 30.01 last 152.74) total: 4500 /unlink in 30.95 seconds: 145.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 761 (time 1787120591.58 total 10.01 last 76.03) - open/close 1537 (time 1787120601.58 total 20.01 last 77.57) - open/close 2149 (time 1787120611.58 total 30.02 last 61.18) - open/close 2903 (time 1787120621.59 total 40.02 last 75.37) - open/close 3536 (time 1787120631.60 total 50.03 last 63.26) - open/close 4280 (time 1787120641.60 total 60.03 last 74.36) total: 4500 open/close in 63.28 seconds: 71.11 ops/second - /unlink 1156 (time 1787120659.77 total 10.00 last 115.60) - /unlink 2297 (time 1787120669.78 total 20.01 last 114.02) - /unlink 3334 (time 1787120679.78 total 30.01 last 103.66) total: 4500 /unlink in 37.91 seconds: 118.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 827 (time 1787120702.42 total 10.02 last 82.57) - open/close 1575 (time 1787120712.42 total 20.02 last 74.76) - open/close 2290 (time 1787120722.44 total 30.04 last 71.36) - open/close 3114 (time 1787120732.45 total 40.05 last 82.31) - open/close 4040 (time 1787120742.46 total 50.05 last 92.56) total: 4500 open/close in 54.71 seconds: 82.25 ops/second - /unlink 1683 (time 1787120761.35 total 10.00 last 168.29) - /unlink 3222 (time 1787120771.36 total 20.02 last 153.66) total: 4500 /unlink in 27.68 seconds: 162.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 864 (time 1787120794.15 total 10.01 last 86.31) - open/close 1770 (time 1787120804.16 total 20.03 last 90.45) - open/close 2699 (time 1787120814.17 total 30.03 last 92.89) - open/close 3639 (time 1787120824.17 total 40.03 last 93.95) total: 4500 open/close in 49.64 seconds: 90.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1494 (time 1787120847.54 total 10.00 last 149.37) - /unlink 3110 (time 1787120857.56 total 20.01 last 161.40) total: 4500 /unlink in 28.52 seconds: 157.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 853 (time 1787120881.02 total 10.00 last 85.29) - open/close 1704 (time 1787120891.02 total 20.01 last 85.05) - open/close 2635 (time 1787120901.02 total 30.01 last 93.09) - open/close 3468 (time 1787120911.03 total 40.01 last 83.25) - open/close 4257 (time 1787120921.04 total 50.02 last 78.85) total: 4500 open/close in 53.48 seconds: 84.15 ops/second - /unlink 1520 (time 1787120938.97 total 10.01 last 151.91) - /unlink 2742 (time 1787120948.97 total 20.01 last 122.11) - /unlink 4309 (time 1787120958.98 total 30.02 last 156.61) total: 4500 /unlink in 31.42 seconds: 143.22 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 880 (time 1787120974.96 total 10.01 last 87.89) - open/close 1815 (time 1787120984.96 total 20.01 last 93.49) - open/close 2643 (time 1787120994.97 total 30.02 last 82.74) - open/close 3523 (time 1787121004.99 total 40.04 last 87.82) - open/close 4224 (time 1787121015.01 total 50.07 last 69.90) total: 4500 open/close in 53.88 seconds: 83.51 ops/second - /unlink 1105 (time 1787121034.33 total 10.01 last 110.41) - /unlink 2711 (time 1787121044.33 total 20.01 last 160.53) - /unlink 4267 (time 1787121054.33 total 30.01 last 155.60) total: 4500 /unlink in 31.97 seconds: 140.78 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11297: 95251 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg146-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg146-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg146-server stop mds service on oleg146-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg146-server stop mds service on oleg146-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg146-server unloading modules via unload_modules_local on: 'oleg146-server' oleg146-server: oleg146-server.virtnet: executing unload_modules_local oleg146-server: modules unloaded.