== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 04:35:52 (1788597352) Stopping clients: oleg314-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg314-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg314-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg314-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg314-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg314-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg314-server oleg314-server: oleg314-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg314-server' oleg314-server: oleg314-server.virtnet: executing load_modules_local oleg314-server: kmemleak disabled oleg314-server: Loading modules from /home/green/git/lustre-release/lustre oleg314-server: detected 4 online CPUs by sysfs oleg314-server: MODOPTS_LIBCFS= oleg314-server: Force libcfs to create 2 CPU partitions oleg314-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg314-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg314-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg314-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg314-server: oleg314-server.virtnet: executing set_default_debug -1 all pdsh@oleg314-client: oleg314-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg314-client.virtnet: -o user_xattr,flock 192.168.203.114@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 858 (time 1788597537.65 total 10.00 last 85.79) - open/close 1751 (time 1788597547.66 total 20.02 last 89.15) - open/close 2630 (time 1788597557.67 total 30.03 last 87.81) - open/close 3431 (time 1788597567.69 total 40.04 last 79.97) - open/close 4301 (time 1788597577.70 total 50.05 last 86.93) total: 4500 open/close in 52.58 seconds: 85.59 ops/second - /unlink 1688 (time 1788597593.87 total 10.01 last 168.66) - /unlink 3215 (time 1788597603.87 total 20.01 last 152.68) total: 4500 /unlink in 28.57 seconds: 157.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 819 (time 1788597626.62 total 10.01 last 81.81) - open/close 1661 (time 1788597636.62 total 20.01 last 84.19) - open/close 2365 (time 1788597646.63 total 30.02 last 70.33) - open/close 3208 (time 1788597656.63 total 40.02 last 84.28) - open/close 4076 (time 1788597666.63 total 50.03 last 86.76) total: 4500 open/close in 55.05 seconds: 81.75 ops/second - /unlink 1728 (time 1788597685.61 total 10.01 last 172.71) - /unlink 3275 (time 1788597695.61 total 20.01 last 154.63) total: 4500 /unlink in 29.80 seconds: 151.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 825 (time 1788597720.27 total 10.00 last 82.47) - open/close 1663 (time 1788597730.28 total 20.01 last 83.76) - open/close 2482 (time 1788597740.28 total 30.01 last 81.86) - open/close 3352 (time 1788597750.28 total 40.02 last 86.98) - open/close 4211 (time 1788597760.29 total 50.02 last 85.88) total: 4500 open/close in 53.74 seconds: 83.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1652 (time 1788597778.25 total 10.00 last 165.16) - /unlink 3227 (time 1788597788.26 total 20.00 last 157.49) total: 4500 /unlink in 27.87 seconds: 161.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 869 (time 1788597811.24 total 10.02 last 86.73) - open/close 1623 (time 1788597821.26 total 20.04 last 75.23) - open/close 2313 (time 1788597831.27 total 30.04 last 68.99) - open/close 3123 (time 1788597841.30 total 40.08 last 80.69) - open/close 3945 (time 1788597851.33 total 50.11 last 81.99) total: 4500 open/close in 57.24 seconds: 78.61 ops/second - /unlink 1534 (time 1788597872.91 total 10.00 last 153.38) - /unlink 3093 (time 1788597882.91 total 20.00 last 155.88) - /unlink 4416 (time 1788597892.91 total 30.00 last 132.27) total: 4500 /unlink in 30.89 seconds: 145.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 563 (time 1788597908.70 total 10.01 last 56.22) - open/close 1187 (time 1788597918.72 total 20.03 last 62.32) - open/close 1794 (time 1788597928.72 total 30.03 last 60.67) - open/close 2545 (time 1788597938.76 total 40.08 last 74.77) - open/close 3343 (time 1788597948.77 total 50.08 last 79.75) - open/close 4195 (time 1788597958.79 total 60.11 last 85.00) total: 4500 open/close in 63.37 seconds: 71.01 ops/second - /unlink 1467 (time 1788597976.35 total 10.01 last 146.60) - /unlink 3003 (time 1788597986.36 total 20.01 last 153.53) total: 4500 /unlink in 29.16 seconds: 154.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 799 (time 1788598009.65 total 10.01 last 79.84) - open/close 1444 (time 1788598019.67 total 20.02 last 64.40) - open/close 2103 (time 1788598029.68 total 30.04 last 65.81) - open/close 2917 (time 1788598039.69 total 40.05 last 81.28) - open/close 3737 (time 1788598049.70 total 50.06 last 81.96) total: 4500 open/close in 59.42 seconds: 75.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1537 (time 1788598072.87 total 10.00 last 153.68) - /unlink 2880 (time 1788598082.87 total 20.00 last 134.25) - /unlink 4323 (time 1788598092.88 total 30.02 last 144.10) total: 4500 /unlink in 31.51 seconds: 142.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 780 (time 1788598108.03 total 10.00 last 77.99) - open/close 1660 (time 1788598118.03 total 20.00 last 88.00) - open/close 2364 (time 1788598128.05 total 30.02 last 70.24) - open/close 3088 (time 1788598138.07 total 40.04 last 72.28) - open/close 3903 (time 1788598148.07 total 50.05 last 81.46) total: 4500 open/close in 57.32 seconds: 78.50 ops/second - /unlink 1471 (time 1788598169.16 total 10.00 last 147.08) - /unlink 3057 (time 1788598179.17 total 20.01 last 158.39) total: 4500 /unlink in 29.60 seconds: 152.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 863 (time 1788598202.87 total 10.00 last 86.28) - open/close 1738 (time 1788598212.88 total 20.01 last 87.46) - open/close 2592 (time 1788598222.88 total 30.01 last 85.39) - open/close 3446 (time 1788598232.88 total 40.01 last 85.38) - open/close 4297 (time 1788598242.89 total 50.02 last 85.01) total: 4500 open/close in 52.53 seconds: 85.67 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1505 (time 1788598259.32 total 10.00 last 150.45) - /unlink 3065 (time 1788598269.33 total 20.01 last 155.86) total: 4500 /unlink in 29.29 seconds: 153.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 752 (time 1788598293.27 total 10.00 last 75.19) - open/close 1483 (time 1788598303.28 total 20.01 last 73.04) - open/close 2187 (time 1788598313.28 total 30.01 last 70.37) - open/close 2898 (time 1788598323.30 total 40.03 last 70.94) - open/close 3707 (time 1788598333.33 total 50.06 last 80.67) - open/close 4453 (time 1788598343.34 total 60.07 last 74.56) total: 4500 open/close in 60.74 seconds: 74.09 ops/second - /unlink 1502 (time 1788598358.05 total 10.00 last 150.14) - /unlink 3081 (time 1788598368.05 total 20.01 last 157.87) total: 4500 /unlink in 29.47 seconds: 152.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 819 (time 1788598391.54 total 10.01 last 81.82) - open/close 1540 (time 1788598401.55 total 20.02 last 72.03) - open/close 2363 (time 1788598411.55 total 30.02 last 82.27) - open/close 3146 (time 1788598421.56 total 40.03 last 78.26) - open/close 4010 (time 1788598431.56 total 50.03 last 86.34) total: 4500 open/close in 55.07 seconds: 81.71 ops/second - /unlink 1597 (time 1788598451.30 total 10.00 last 159.67) - /unlink 3241 (time 1788598461.31 total 20.02 last 164.19) total: 4500 /unlink in 28.61 seconds: 157.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 769 (time 1788598484.43 total 10.00 last 76.90) - open/close 1388 (time 1788598494.45 total 20.02 last 61.79) - open/close 2071 (time 1788598504.45 total 30.02 last 68.27) - open/close 2924 (time 1788598514.46 total 40.03 last 85.23) - open/close 3667 (time 1788598524.48 total 50.04 last 74.19) - open/close 4470 (time 1788598534.48 total 60.04 last 80.30) total: 4500 open/close in 60.31 seconds: 74.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1410 (time 1788598548.79 total 10.00 last 140.95) - /unlink 3044 (time 1788598558.79 total 20.00 last 163.40) total: 4500 /unlink in 29.70 seconds: 151.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 813 (time 1788598582.64 total 10.00 last 81.28) - open/close 1610 (time 1788598592.64 total 20.01 last 79.68) - open/close 2478 (time 1788598602.65 total 30.01 last 86.80) - open/close 3264 (time 1788598612.65 total 40.01 last 78.58) - open/close 4081 (time 1788598622.65 total 50.01 last 81.65) total: 4500 open/close in 54.69 seconds: 82.27 ops/second - /unlink 1327 (time 1788598643.06 total 10.01 last 132.55) - /unlink 2591 (time 1788598653.07 total 20.02 last 126.24) - /unlink 4050 (time 1788598663.08 total 30.03 last 145.85) total: 4500 /unlink in 32.92 seconds: 136.68 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 770 (time 1788598681.06 total 10.01 last 76.95) - open/close 1436 (time 1788598691.07 total 20.02 last 66.52) - open/close 2155 (time 1788598701.08 total 30.02 last 71.87) - open/close 2908 (time 1788598711.08 total 40.03 last 75.26) - open/close 3719 (time 1788598721.09 total 50.03 last 81.10) - open/close 4491 (time 1788598731.09 total 60.04 last 77.14) total: 4500 open/close in 60.12 seconds: 74.84 ops/second - /unlink 1462 (time 1788598745.80 total 10.00 last 146.20) - /unlink 2989 (time 1788598755.80 total 20.00 last 152.69) total: 4500 /unlink in 29.60 seconds: 152.04 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 888 (time 1788598779.34 total 10.02 last 88.64) - open/close 1714 (time 1788598789.34 total 20.02 last 82.57) - open/close 2558 (time 1788598799.34 total 30.03 last 84.36) - open/close 3365 (time 1788598809.35 total 40.03 last 80.65) - open/close 4210 (time 1788598819.35 total 50.03 last 84.50) total: 4500 open/close in 54.82 seconds: 82.08 ops/second - /unlink 1602 (time 1788598838.81 total 10.00 last 160.20) - /unlink 3006 (time 1788598848.82 total 20.00 last 140.36) total: 4500 /unlink in 29.90 seconds: 150.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 735 (time 1788598873.62 total 10.02 last 73.37) - open/close 1502 (time 1788598883.63 total 20.03 last 76.61) - open/close 2310 (time 1788598893.64 total 30.04 last 80.75) - open/close 3071 (time 1788598903.65 total 40.04 last 76.05) - open/close 3843 (time 1788598913.65 total 50.05 last 77.17) total: 4500 open/close in 57.87 seconds: 77.76 ops/second - /unlink 1473 (time 1788598935.97 total 10.01 last 147.15) - /unlink 3000 (time 1788598945.98 total 20.02 last 152.60) - /unlink 4347 (time 1788598955.99 total 30.03 last 134.52) total: 4500 /unlink in 31.08 seconds: 144.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 765 (time 1788598971.79 total 10.00 last 76.46) - open/close 1505 (time 1788598981.81 total 20.03 last 73.82) - open/close 2179 (time 1788598991.82 total 30.03 last 67.37) - open/close 2973 (time 1788599001.82 total 40.04 last 79.38) - open/close 3720 (time 1788599011.82 total 50.04 last 74.68) - open/close 4482 (time 1788599021.83 total 60.05 last 76.14) total: 4500 open/close in 60.33 seconds: 74.58 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1461 (time 1788599037.08 total 10.00 last 146.08) - /unlink 2768 (time 1788599047.09 total 20.01 last 130.59) - /unlink 4173 (time 1788599057.10 total 30.02 last 140.38) total: 4500 /unlink in 32.75 seconds: 137.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 711 (time 1788599074.72 total 10.01 last 71.03) - open/close 1388 (time 1788599084.73 total 20.01 last 67.68) - open/close 2146 (time 1788599094.76 total 30.05 last 75.54) - open/close 2905 (time 1788599104.77 total 40.05 last 75.85) - open/close 3602 (time 1788599114.77 total 50.06 last 69.65) - open/close 4339 (time 1788599124.78 total 60.06 last 73.69) total: 4500 open/close in 61.94 seconds: 72.66 ops/second - /unlink 1391 (time 1788599140.76 total 10.01 last 138.91) - /unlink 2765 (time 1788599150.76 total 20.01 last 137.39) - /unlink 4238 (time 1788599160.76 total 30.02 last 147.25) total: 4500 /unlink in 32.00 seconds: 140.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 650 (time 1788599177.06 total 10.00 last 64.98) - open/close 1386 (time 1788599187.06 total 20.01 last 73.57) - open/close 2032 (time 1788599197.07 total 30.02 last 64.53) - open/close 2804 (time 1788599207.09 total 40.03 last 77.10) - open/close 3515 (time 1788599217.11 total 50.06 last 70.92) - open/close 4209 (time 1788599227.12 total 60.06 last 69.37) total: 4500 open/close in 64.69 seconds: 69.57 ops/second - /unlink 1323 (time 1788599247.26 total 10.01 last 132.11) - /unlink 2724 (time 1788599257.26 total 20.02 last 139.99) - /unlink 4072 (time 1788599267.27 total 30.03 last 134.70) total: 4500 /unlink in 33.61 seconds: 133.89 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11344: 94999 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg314-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg314-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg314-server stop mds service on oleg314-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg314-server stop mds service on oleg314-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg314-server unloading modules via unload_modules_local on: 'oleg314-server' oleg314-server: oleg314-server.virtnet: executing unload_modules_local oleg314-server: kmemleak disabled modules unloaded.