== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 09:19:03 (1789391943) Stopping clients: oleg112-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg112-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg112-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg112-server oleg112-server: oleg112-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing load_modules_local oleg112-server: kmemleak disabled oleg112-server: Loading modules from /home/green/git/lustre-release/lustre oleg112-server: detected 4 online CPUs by sysfs oleg112-server: MODOPTS_LIBCFS= oleg112-server: Force libcfs to create 2 CPU partitions oleg112-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg112-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 start mds service on oleg112-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg112-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg112-server: oleg112-server.virtnet: executing set_default_debug -1 all pdsh@oleg112-client: oleg112-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg112-client.virtnet: -o user_xattr,flock 192.168.201.112@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 936 (time 1789392148.64 total 10.00 last 93.57) - open/close 1765 (time 1789392158.65 total 20.01 last 82.86) - open/close 2633 (time 1789392168.66 total 30.02 last 86.72) - open/close 3502 (time 1789392178.66 total 40.02 last 86.87) - open/close 4448 (time 1789392188.67 total 50.03 last 94.49) total: 4500 open/close in 50.72 seconds: 88.73 ops/second - /unlink 1600 (time 1789392204.32 total 10.01 last 159.84) - /unlink 2840 (time 1789392214.32 total 20.01 last 123.96) - /unlink 4191 (time 1789392224.32 total 30.02 last 135.05) total: 4500 /unlink in 32.52 seconds: 138.38 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 793 (time 1789392242.08 total 10.00 last 79.28) - open/close 1579 (time 1789392252.08 total 20.01 last 78.56) - open/close 2356 (time 1789392262.10 total 30.02 last 77.60) - open/close 3298 (time 1789392272.10 total 40.02 last 94.18) - open/close 4263 (time 1789392282.10 total 50.02 last 96.49) total: 4500 open/close in 52.47 seconds: 85.77 ops/second - /unlink 1063 (time 1789392299.10 total 10.00 last 106.25) - /unlink 2463 (time 1789392309.11 total 20.01 last 139.95) - /unlink 4020 (time 1789392319.11 total 30.01 last 155.68) total: 4500 /unlink in 33.36 seconds: 134.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 918 (time 1789392337.27 total 10.01 last 91.73) - open/close 1757 (time 1789392347.28 total 20.01 last 83.85) - open/close 2639 (time 1789392357.28 total 30.02 last 88.17) - open/close 3556 (time 1789392367.29 total 40.03 last 91.60) - open/close 4347 (time 1789392377.30 total 50.03 last 79.05) total: 4500 open/close in 51.62 seconds: 87.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1683 (time 1789392393.84 total 10.00 last 168.23) - /unlink 3278 (time 1789392403.85 total 20.01 last 159.44) total: 4500 /unlink in 27.44 seconds: 164.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 864 (time 1789392425.81 total 10.02 last 86.26) - open/close 1642 (time 1789392435.82 total 20.02 last 77.79) - open/close 2430 (time 1789392445.83 total 30.03 last 78.70) - open/close 3201 (time 1789392455.84 total 40.04 last 77.04) - open/close 3910 (time 1789392465.84 total 50.04 last 70.89) total: 4500 open/close in 57.82 seconds: 77.83 ops/second - /unlink 1926 (time 1789392490.42 total 10.00 last 192.54) - /unlink 3263 (time 1789392500.42 total 20.01 last 133.67) total: 4500 /unlink in 29.68 seconds: 151.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 876 (time 1789392526.04 total 10.01 last 87.54) - open/close 1732 (time 1789392536.04 total 20.01 last 85.58) - open/close 2667 (time 1789392546.05 total 30.02 last 93.42) - open/close 3572 (time 1789392556.06 total 40.03 last 90.41) - open/close 4361 (time 1789392566.07 total 50.03 last 78.86) total: 4500 open/close in 51.54 seconds: 87.30 ops/second - /unlink 1722 (time 1789392582.33 total 10.00 last 172.15) - /unlink 3168 (time 1789392592.33 total 20.01 last 144.56) total: 4500 /unlink in 29.48 seconds: 152.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 762 (time 1789392616.09 total 10.01 last 76.12) - open/close 1538 (time 1789392626.10 total 20.03 last 77.47) - open/close 2436 (time 1789392636.11 total 30.03 last 89.77) - open/close 3143 (time 1789392646.11 total 40.04 last 70.66) - open/close 3964 (time 1789392656.12 total 50.04 last 82.05) total: 4500 open/close in 57.30 seconds: 78.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1342 (time 1789392678.99 total 10.00 last 134.15) - /unlink 2769 (time 1789392688.99 total 20.01 last 142.65) - /unlink 4358 (time 1789392699.00 total 30.01 last 158.85) total: 4500 /unlink in 31.14 seconds: 144.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 812 (time 1789392716.05 total 10.01 last 81.11) - open/close 1569 (time 1789392726.06 total 20.01 last 75.67) - open/close 2346 (time 1789392736.08 total 30.03 last 77.54) - open/close 3113 (time 1789392746.08 total 40.04 last 76.69) - open/close 3857 (time 1789392756.09 total 50.04 last 74.35) total: 4500 open/close in 58.08 seconds: 77.48 ops/second - /unlink 1622 (time 1789392778.73 total 10.00 last 162.18) - /unlink 3070 (time 1789392788.73 total 20.01 last 144.72) - /unlink 4391 (time 1789392798.73 total 30.01 last 132.10) total: 4500 /unlink in 30.93 seconds: 145.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 740 (time 1789392814.01 total 10.00 last 73.98) - open/close 1579 (time 1789392824.02 total 20.00 last 83.88) - open/close 2429 (time 1789392834.02 total 30.01 last 84.96) - open/close 3337 (time 1789392844.03 total 40.02 last 90.73) - open/close 4217 (time 1789392854.04 total 50.03 last 87.90) total: 4500 open/close in 53.49 seconds: 84.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1572 (time 1789392873.39 total 10.00 last 157.15) - /unlink 3193 (time 1789392883.39 total 20.00 last 162.09) total: 4500 /unlink in 29.50 seconds: 152.56 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 701 (time 1789392908.77 total 10.02 last 69.97) - open/close 1403 (time 1789392918.79 total 20.03 last 70.10) - open/close 2101 (time 1789392928.79 total 30.04 last 69.78) - open/close 2962 (time 1789392938.80 total 40.04 last 86.05) - open/close 3876 (time 1789392948.81 total 50.05 last 91.29) total: 4500 open/close in 58.51 seconds: 76.91 ops/second - /unlink 1498 (time 1789392972.12 total 10.01 last 149.61) - /unlink 2631 (time 1789392982.12 total 20.02 last 113.22) - /unlink 3910 (time 1789392992.13 total 30.02 last 127.88) total: 4500 /unlink in 33.54 seconds: 134.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 651 (time 1789393011.34 total 10.00 last 65.08) - open/close 1266 (time 1789393021.35 total 20.02 last 61.42) - open/close 2167 (time 1789393031.36 total 30.02 last 90.04) - open/close 2972 (time 1789393041.37 total 40.04 last 80.40) - open/close 3802 (time 1789393051.38 total 50.04 last 82.93) total: 4500 open/close in 58.58 seconds: 76.82 ops/second - /unlink 1243 (time 1789393075.00 total 10.00 last 124.25) - /unlink 2623 (time 1789393085.00 total 20.01 last 137.97) - /unlink 4061 (time 1789393095.02 total 30.02 last 143.64) total: 4500 /unlink in 33.17 seconds: 135.67 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 740 (time 1789393113.27 total 10.00 last 73.98) - open/close 1577 (time 1789393123.27 total 20.00 last 83.69) - open/close 2412 (time 1789393133.28 total 30.02 last 83.39) - open/close 3371 (time 1789393143.29 total 40.02 last 95.88) - open/close 4287 (time 1789393153.29 total 50.02 last 91.57) total: 4500 open/close in 52.47 seconds: 85.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1681 (time 1789393170.21 total 10.00 last 168.08) - /unlink 3342 (time 1789393180.21 total 20.00 last 166.09) total: 4500 /unlink in 26.83 seconds: 167.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 675 (time 1789393201.57 total 10.01 last 67.44) - open/close 1578 (time 1789393211.58 total 20.02 last 90.21) - open/close 2479 (time 1789393221.58 total 30.02 last 90.07) - open/close 3394 (time 1789393231.58 total 40.02 last 91.50) - open/close 4335 (time 1789393241.60 total 50.04 last 93.97) total: 4500 open/close in 52.01 seconds: 86.51 ops/second - /unlink 1523 (time 1789393257.91 total 10.00 last 152.30) - /unlink 3229 (time 1789393267.92 total 20.00 last 170.54) total: 4500 /unlink in 28.09 seconds: 160.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 779 (time 1789393290.18 total 10.01 last 77.84) - open/close 1632 (time 1789393300.18 total 20.02 last 85.23) - open/close 2463 (time 1789393310.20 total 30.03 last 82.96) - open/close 3353 (time 1789393320.21 total 40.04 last 88.93) - open/close 4135 (time 1789393330.23 total 50.06 last 78.06) total: 4500 open/close in 55.91 seconds: 80.48 ops/second - /unlink 1681 (time 1789393350.31 total 10.00 last 168.08) - /unlink 3326 (time 1789393360.31 total 20.00 last 164.48) total: 4500 /unlink in 27.87 seconds: 161.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 797 (time 1789393383.15 total 10.01 last 79.64) - open/close 1653 (time 1789393393.16 total 20.02 last 85.49) - open/close 2587 (time 1789393403.17 total 30.02 last 93.38) - open/close 3494 (time 1789393413.17 total 40.03 last 90.65) - open/close 4435 (time 1789393423.17 total 50.03 last 94.09) total: 4500 open/close in 50.85 seconds: 88.50 ops/second - /unlink 1581 (time 1789393438.01 total 10.00 last 158.10) - /unlink 3281 (time 1789393448.01 total 20.00 last 170.00) total: 4500 /unlink in 29.89 seconds: 150.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 898 (time 1789393472.38 total 10.01 last 89.74) - open/close 1834 (time 1789393482.39 total 20.01 last 93.56) - open/close 2870 (time 1789393492.40 total 30.02 last 103.51) - open/close 3751 (time 1789393502.40 total 40.03 last 88.04) total: 4500 open/close in 48.96 seconds: 91.91 ops/second - /unlink 1581 (time 1789393525.44 total 10.00 last 158.05) - /unlink 3270 (time 1789393535.44 total 20.01 last 168.78) total: 4500 /unlink in 26.69 seconds: 168.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 831 (time 1789393556.15 total 10.01 last 83.03) - open/close 1689 (time 1789393566.17 total 20.02 last 85.68) - open/close 2513 (time 1789393576.17 total 30.02 last 82.39) - open/close 3298 (time 1789393586.18 total 40.03 last 78.43) - open/close 4294 (time 1789393596.18 total 50.04 last 99.56) total: 4500 open/close in 52.11 seconds: 86.36 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1512 (time 1789393612.89 total 10.00 last 151.18) - /unlink 3174 (time 1789393622.89 total 20.00 last 166.17) total: 4500 /unlink in 28.60 seconds: 157.36 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 862 (time 1789393646.55 total 10.04 last 85.82) - open/close 1775 (time 1789393656.56 total 20.05 last 91.26) - open/close 2704 (time 1789393666.57 total 30.06 last 92.83) - open/close 3654 (time 1789393676.57 total 40.06 last 95.00) - open/close 4391 (time 1789393686.57 total 50.06 last 73.67) total: 4500 open/close in 51.59 seconds: 87.22 ops/second - /unlink 1760 (time 1789393703.07 total 10.00 last 175.92) - /unlink 3547 (time 1789393713.07 total 20.00 last 178.70) total: 4500 /unlink in 26.47 seconds: 170.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 849 (time 1789393733.51 total 10.00 last 84.90) - open/close 1623 (time 1789393743.51 total 20.00 last 77.38) - open/close 2468 (time 1789393753.53 total 30.02 last 84.38) - open/close 3336 (time 1789393763.53 total 40.02 last 86.75) - open/close 4233 (time 1789393773.54 total 50.03 last 89.61) total: 4500 open/close in 53.26 seconds: 84.49 ops/second - /unlink 1734 (time 1789393791.19 total 10.01 last 173.27) - /unlink 3194 (time 1789393801.19 total 20.01 last 145.97) total: 4500 /unlink in 27.91 seconds: 161.24 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11415: 96153 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg112-client.virtnet /mnt/lustre (opts:) oleg112-server: params: OBD_IOC_LLOG_PRINT failed: No such file or directory pdsh@oleg112-client: oleg112-server: ssh exited with exit code 2 pdsh@oleg112-client: oleg112-server: ssh exited with exit code 123 allow_register=unlimited stop ost1 service on oleg112-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg112-server stop mds service on oleg112-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg112-server unloading modules via unload_modules_local on: 'oleg112-server' oleg112-server: oleg112-server.virtnet: executing unload_modules_local oleg112-server: kmemleak disabled modules unloaded.