== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 19:49:53 (1787788193) Stopping clients: oleg307-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg307-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg307-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg307-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg307-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg307-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg307-server oleg307-server: oleg307-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg307-server' oleg307-server: oleg307-server.virtnet: executing load_modules_local oleg307-server: Loading modules from /home/green/git/lustre-release/lustre oleg307-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg307-server: detected 4 online CPUs by sysfs oleg307-server: MODOPTS_LIBCFS= oleg307-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg307-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg307-server: oleg307-server.virtnet: executing set_default_debug -1 all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg307-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg307-server: oleg307-server.virtnet: executing set_default_debug -1 all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg307-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg307-server: oleg307-server.virtnet: executing set_default_debug -1 all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg307-client.virtnet: -o user_xattr,flock 192.168.203.107@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 846 (time 1787788371.17 total 10.01 last 84.53) - open/close 1650 (time 1787788381.17 total 20.01 last 80.40) - open/close 2487 (time 1787788391.17 total 30.01 last 83.69) - open/close 3266 (time 1787788401.18 total 40.01 last 77.88) - open/close 4028 (time 1787788411.18 total 50.02 last 76.17) total: 4500 open/close in 56.41 seconds: 79.77 ops/second - /unlink 1527 (time 1787788431.88 total 10.00 last 152.68) - /unlink 3128 (time 1787788441.89 total 20.01 last 159.94) total: 4500 /unlink in 29.06 seconds: 154.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 821 (time 1787788465.34 total 10.01 last 82.02) - open/close 1644 (time 1787788475.34 total 20.01 last 82.27) - open/close 2507 (time 1787788485.34 total 30.02 last 86.29) - open/close 3367 (time 1787788495.36 total 40.03 last 85.84) - open/close 4113 (time 1787788505.37 total 50.04 last 74.55) total: 4500 open/close in 55.24 seconds: 81.47 ops/second - /unlink 1375 (time 1787788525.74 total 10.00 last 137.47) - /unlink 2925 (time 1787788535.74 total 20.00 last 154.97) - /unlink 4490 (time 1787788545.74 total 30.00 last 156.50) total: 4500 /unlink in 30.06 seconds: 149.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 822 (time 1787788559.36 total 10.01 last 82.14) - open/close 1669 (time 1787788569.36 total 20.01 last 84.69) - open/close 2536 (time 1787788579.37 total 30.01 last 86.65) - open/close 3319 (time 1787788589.37 total 40.02 last 78.28) - open/close 4157 (time 1787788599.37 total 50.02 last 83.76) total: 4500 open/close in 53.93 seconds: 83.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1478 (time 1787788616.73 total 10.02 last 147.53) - /unlink 3072 (time 1787788626.74 total 20.03 last 159.26) total: 4500 /unlink in 28.59 seconds: 157.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 848 (time 1787788649.50 total 10.00 last 84.77) - open/close 1617 (time 1787788659.51 total 20.01 last 76.84) - open/close 2443 (time 1787788669.51 total 30.01 last 82.57) - open/close 3291 (time 1787788679.53 total 40.03 last 84.69) - open/close 4127 (time 1787788689.54 total 50.04 last 83.47) total: 4500 open/close in 54.16 seconds: 83.09 ops/second - /unlink 1616 (time 1787788707.44 total 10.00 last 161.54) - /unlink 3168 (time 1787788717.44 total 20.00 last 155.18) total: 4500 /unlink in 28.63 seconds: 157.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 827 (time 1787788739.95 total 10.00 last 82.68) - open/close 1638 (time 1787788749.96 total 20.01 last 81.04) - open/close 2482 (time 1787788759.96 total 30.01 last 84.39) - open/close 3313 (time 1787788769.97 total 40.02 last 83.02) - open/close 4135 (time 1787788779.97 total 50.02 last 82.17) total: 4500 open/close in 54.45 seconds: 82.65 ops/second - /unlink 1571 (time 1787788798.33 total 10.00 last 157.09) - /unlink 3133 (time 1787788808.34 total 20.00 last 156.19) total: 4500 /unlink in 28.30 seconds: 159.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 808 (time 1787788831.00 total 10.01 last 80.74) - open/close 1677 (time 1787788841.00 total 20.01 last 86.86) - open/close 2523 (time 1787788851.01 total 30.02 last 84.55) - open/close 3366 (time 1787788861.01 total 40.02 last 84.30) - open/close 4237 (time 1787788871.01 total 50.02 last 87.07) total: 4500 open/close in 52.89 seconds: 85.07 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1648 (time 1787788887.11 total 10.00 last 164.78) - /unlink 3262 (time 1787788897.11 total 20.01 last 161.33) total: 4500 /unlink in 27.41 seconds: 164.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 879 (time 1787788918.55 total 10.00 last 87.89) - open/close 1743 (time 1787788928.56 total 20.00 last 86.38) - open/close 2641 (time 1787788938.56 total 30.00 last 89.79) - open/close 3448 (time 1787788948.57 total 40.02 last 80.60) - open/close 4040 (time 1787788958.57 total 50.02 last 59.19) total: 4500 open/close in 56.99 seconds: 78.97 ops/second - /unlink 1474 (time 1787788979.75 total 10.00 last 147.38) - /unlink 3103 (time 1787788989.75 total 20.00 last 162.86) total: 4500 /unlink in 28.87 seconds: 155.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 852 (time 1787789012.65 total 10.00 last 85.20) - open/close 1656 (time 1787789022.65 total 20.00 last 80.38) - open/close 2544 (time 1787789032.68 total 30.03 last 88.60) - open/close 3376 (time 1787789042.70 total 40.05 last 83.00) - open/close 4212 (time 1787789052.71 total 50.06 last 83.48) total: 4500 open/close in 53.85 seconds: 83.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1595 (time 1787789070.46 total 10.00 last 159.45) - /unlink 3108 (time 1787789080.47 total 20.01 last 151.21) total: 4500 /unlink in 29.11 seconds: 154.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 849 (time 1787789103.74 total 10.00 last 84.89) - open/close 1666 (time 1787789113.75 total 20.01 last 81.66) - open/close 2618 (time 1787789123.76 total 30.01 last 95.11) - open/close 3599 (time 1787789133.76 total 40.02 last 98.09) total: 4500 open/close in 49.91 seconds: 90.16 ops/second - /unlink 1651 (time 1787789157.11 total 10.01 last 164.99) - /unlink 3316 (time 1787789167.11 total 20.01 last 166.48) total: 4500 /unlink in 27.48 seconds: 163.75 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 817 (time 1787789188.15 total 10.01 last 81.64) - open/close 1630 (time 1787789198.16 total 20.02 last 81.23) - open/close 2504 (time 1787789208.17 total 30.03 last 87.29) - open/close 3409 (time 1787789218.18 total 40.03 last 90.47) - open/close 4229 (time 1787789228.18 total 50.04 last 81.93) total: 4500 open/close in 53.94 seconds: 83.42 ops/second - /unlink 1585 (time 1787789245.71 total 10.01 last 158.40) - /unlink 3132 (time 1787789255.71 total 20.01 last 154.63) total: 4500 /unlink in 27.88 seconds: 161.38 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 940 (time 1787789276.75 total 10.01 last 93.91) - open/close 1690 (time 1787789286.77 total 20.02 last 74.90) - open/close 2541 (time 1787789296.77 total 30.03 last 85.07) - open/close 3317 (time 1787789306.78 total 40.04 last 77.53) - open/close 4219 (time 1787789316.79 total 50.05 last 90.07) total: 4500 open/close in 52.93 seconds: 85.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1484 (time 1787789333.98 total 10.01 last 148.28) - /unlink 3112 (time 1787789343.98 total 20.01 last 162.79) total: 4500 /unlink in 27.94 seconds: 161.04 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 995 (time 1787789364.78 total 10.01 last 99.42) - open/close 1942 (time 1787789374.79 total 20.02 last 94.62) - open/close 2867 (time 1787789384.80 total 30.02 last 92.48) - open/close 3805 (time 1787789394.81 total 40.03 last 93.69) total: 4500 open/close in 47.99 seconds: 93.78 ops/second - /unlink 1768 (time 1787789415.83 total 10.00 last 176.73) - /unlink 3511 (time 1787789425.84 total 20.02 last 174.06) total: 4500 /unlink in 25.32 seconds: 177.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 923 (time 1787789444.02 total 10.01 last 92.24) - open/close 1803 (time 1787789454.02 total 20.01 last 87.95) - open/close 2700 (time 1787789464.03 total 30.02 last 89.61) - open/close 3676 (time 1787789474.04 total 40.03 last 97.55) total: 4500 open/close in 49.28 seconds: 91.31 ops/second - /unlink 1720 (time 1787789496.50 total 10.00 last 171.95) - /unlink 3475 (time 1787789506.50 total 20.00 last 175.48) total: 4500 /unlink in 25.86 seconds: 174.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 872 (time 1787789525.98 total 10.00 last 87.16) - open/close 1791 (time 1787789535.99 total 20.02 last 91.80) - open/close 2778 (time 1787789546.00 total 30.03 last 98.60) - open/close 3794 (time 1787789556.01 total 40.04 last 101.49) total: 4500 open/close in 47.20 seconds: 95.33 ops/second - /unlink 1721 (time 1787789575.98 total 10.00 last 172.05) - /unlink 3450 (time 1787789585.98 total 20.00 last 172.90) total: 4500 /unlink in 26.92 seconds: 167.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 939 (time 1787789605.90 total 10.01 last 93.81) - open/close 1842 (time 1787789615.91 total 20.02 last 90.22) - open/close 2690 (time 1787789625.92 total 30.03 last 84.73) - open/close 3547 (time 1787789635.92 total 40.03 last 85.67) total: 4500 open/close in 49.84 seconds: 90.29 ops/second - /unlink 1596 (time 1787789659.07 total 10.00 last 159.60) - /unlink 3202 (time 1787789669.08 total 20.00 last 160.54) total: 4500 /unlink in 28.04 seconds: 160.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 988 (time 1787789691.89 total 10.00 last 98.79) - open/close 1904 (time 1787789701.89 total 20.01 last 91.56) - open/close 2789 (time 1787789711.89 total 30.01 last 88.47) - open/close 3597 (time 1787789721.90 total 40.02 last 80.75) - open/close 4336 (time 1787789731.90 total 50.02 last 73.88) total: 4500 open/close in 51.77 seconds: 86.93 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1529 (time 1787789746.66 total 10.00 last 152.90) - /unlink 3132 (time 1787789756.67 total 20.01 last 160.19) total: 4500 /unlink in 29.48 seconds: 152.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 838 (time 1787789779.35 total 10.03 last 83.57) - open/close 1717 (time 1787789789.35 total 20.03 last 87.87) - open/close 2619 (time 1787789799.36 total 30.03 last 90.18) - open/close 3480 (time 1787789809.36 total 40.04 last 86.05) - open/close 4349 (time 1787789819.36 total 50.04 last 86.89) total: 4500 open/close in 51.60 seconds: 87.22 ops/second - /unlink 1679 (time 1787789834.95 total 10.00 last 167.89) - /unlink 3229 (time 1787789844.95 total 20.00 last 154.94) total: 4500 /unlink in 27.09 seconds: 166.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 961 (time 1787789864.93 total 10.00 last 96.08) - open/close 1955 (time 1787789874.96 total 20.04 last 99.04) - open/close 2891 (time 1787789884.98 total 30.05 last 93.49) - open/close 3911 (time 1787789894.99 total 40.06 last 101.90) total: 4500 open/close in 46.25 seconds: 97.29 ops/second - /unlink 1971 (time 1787789913.80 total 10.00 last 197.10) - /unlink 3751 (time 1787789923.80 total 20.00 last 177.92) total: 4500 /unlink in 23.84 seconds: 188.79 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95142 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg307-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg307-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg307-server stop mds service on oleg307-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg307-server stop mds service on oleg307-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg307-server unloading modules via unload_modules_local on: 'oleg307-server' oleg307-server: oleg307-server.virtnet: executing unload_modules_local oleg307-server: modules unloaded.