== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 03:21:17 (1780384877) Stopping clients: oleg455-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg455-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg455-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg455-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg455-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg455-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg455-server oleg455-server: oleg455-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg455-server' oleg455-server: oleg455-server.virtnet: executing load_modules_local oleg455-server: Loading modules from /home/green/git/lustre-release/lustre oleg455-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg455-server: detected 4 online CPUs by sysfs oleg455-server: MODOPTS_LIBCFS= oleg455-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg455-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg455-server: oleg455-server.virtnet: executing set_default_debug -1 all pdsh@oleg455-client: oleg455-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg455-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg455-server: oleg455-server.virtnet: executing set_default_debug -1 all pdsh@oleg455-client: oleg455-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg455-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg455-server: oleg455-server.virtnet: executing set_default_debug -1 all pdsh@oleg455-client: oleg455-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg455-client.virtnet: -o user_xattr,flock 192.168.204.155@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 759 (time 1780385043.98 total 10.01 last 75.83) - open/close 1543 (time 1780385053.98 total 20.01 last 78.36) - open/close 2359 (time 1780385063.99 total 30.03 last 81.51) - open/close 3188 (time 1780385074.00 total 40.03 last 82.83) - open/close 4021 (time 1780385084.01 total 50.04 last 83.22) total: 4500 open/close in 55.72 seconds: 80.76 ops/second - /unlink 1337 (time 1780385103.04 total 10.00 last 133.64) - /unlink 2782 (time 1780385113.04 total 20.01 last 144.48) - /unlink 4276 (time 1780385123.04 total 30.01 last 149.38) total: 4500 /unlink in 31.57 seconds: 142.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 919 (time 1780385137.69 total 10.00 last 91.86) - open/close 1745 (time 1780385147.70 total 20.01 last 82.54) - open/close 2554 (time 1780385157.70 total 30.02 last 80.85) - open/close 3328 (time 1780385167.71 total 40.02 last 77.36) - open/close 4103 (time 1780385177.72 total 50.03 last 77.43) total: 4500 open/close in 55.03 seconds: 81.78 ops/second - /unlink 1577 (time 1780385196.09 total 10.01 last 157.60) - /unlink 3043 (time 1780385206.10 total 20.01 last 146.53) total: 4500 /unlink in 29.48 seconds: 152.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 814 (time 1780385228.62 total 10.00 last 81.36) - open/close 1638 (time 1780385238.63 total 20.01 last 82.32) - open/close 2589 (time 1780385248.63 total 30.02 last 95.05) - open/close 3400 (time 1780385258.64 total 40.03 last 81.01) - open/close 4209 (time 1780385268.65 total 50.04 last 80.83) total: 4500 open/close in 53.58 seconds: 83.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1819 (time 1780385285.07 total 10.00 last 181.85) - /unlink 3731 (time 1780385295.07 total 20.01 last 191.12) total: 4500 /unlink in 24.19 seconds: 186.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 906 (time 1780385312.33 total 10.00 last 90.56) - open/close 1866 (time 1780385322.34 total 20.02 last 95.87) - open/close 2789 (time 1780385332.34 total 30.02 last 92.27) - open/close 3715 (time 1780385342.34 total 40.02 last 92.59) - open/close 4432 (time 1780385352.36 total 50.04 last 71.56) total: 4500 open/close in 51.39 seconds: 87.56 ops/second - /unlink 998 (time 1780385370.08 total 10.00 last 99.80) - /unlink 1847 (time 1780385380.08 total 20.00 last 84.88) - /unlink 2734 (time 1780385390.08 total 30.01 last 88.67) - /unlink 3871 (time 1780385400.09 total 40.01 last 113.66) total: 4500 /unlink in 45.16 seconds: 99.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 664 (time 1780385419.56 total 10.02 last 66.29) - open/close 1370 (time 1780385429.56 total 20.02 last 70.55) - open/close 2098 (time 1780385439.57 total 30.03 last 72.74) - open/close 2762 (time 1780385449.58 total 40.04 last 66.35) - open/close 3450 (time 1780385459.59 total 50.05 last 68.71) - open/close 4081 (time 1780385469.61 total 60.07 last 63.00) total: 4500 open/close in 66.42 seconds: 67.75 ops/second - /unlink 1314 (time 1780385490.11 total 10.00 last 131.35) - /unlink 2607 (time 1780385500.11 total 20.01 last 129.26) - /unlink 3863 (time 1780385510.12 total 30.01 last 125.53) total: 4500 /unlink in 35.33 seconds: 127.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 617 (time 1780385530.45 total 10.02 last 61.57) - open/close 1241 (time 1780385540.46 total 20.03 last 62.33) - open/close 1761 (time 1780385550.48 total 30.05 last 51.90) - open/close 2360 (time 1780385560.50 total 40.06 last 59.83) - open/close 3041 (time 1780385570.50 total 50.07 last 68.06) - open/close 3724 (time 1780385580.51 total 60.08 last 68.22) - open/close 4402 (time 1780385590.52 total 70.09 last 67.72) total: 4500 open/close in 71.59 seconds: 62.86 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1220 (time 1780385606.68 total 10.01 last 121.90) - /unlink 2395 (time 1780385616.68 total 20.01 last 117.45) - /unlink 3677 (time 1780385626.68 total 30.01 last 128.18) total: 4500 /unlink in 36.24 seconds: 124.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 593 (time 1780385648.04 total 10.02 last 59.18) - open/close 1130 (time 1780385658.05 total 20.03 last 53.62) - open/close 1734 (time 1780385668.07 total 30.05 last 60.29) - open/close 2325 (time 1780385678.10 total 40.08 last 58.96) - open/close 2980 (time 1780385688.10 total 50.08 last 65.46) - open/close 3694 (time 1780385698.11 total 60.09 last 71.35) - open/close 4452 (time 1780385708.11 total 70.09 last 75.79) total: 4500 open/close in 70.73 seconds: 63.62 ops/second - /unlink 1386 (time 1780385723.21 total 10.01 last 138.52) - /unlink 2661 (time 1780385733.21 total 20.01 last 127.45) - /unlink 4083 (time 1780385743.22 total 30.01 last 142.14) total: 4500 /unlink in 33.36 seconds: 134.90 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 708 (time 1780385760.57 total 10.01 last 70.73) - open/close 1386 (time 1780385770.59 total 20.03 last 67.68) - open/close 1979 (time 1780385780.60 total 30.03 last 59.27) - open/close 2684 (time 1780385790.60 total 40.04 last 70.47) - open/close 3492 (time 1780385800.61 total 50.05 last 80.72) - open/close 4231 (time 1780385810.61 total 60.05 last 73.90) total: 4500 open/close in 63.64 seconds: 70.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1442 (time 1780385828.17 total 10.00 last 144.13) - /unlink 2918 (time 1780385838.17 total 20.01 last 147.58) total: 4500 /unlink in 29.93 seconds: 150.36 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 755 (time 1780385861.25 total 10.00 last 75.48) - open/close 1613 (time 1780385871.27 total 20.02 last 85.67) - open/close 2535 (time 1780385881.27 total 30.02 last 92.16) - open/close 3381 (time 1780385891.28 total 40.03 last 84.52) - open/close 4203 (time 1780385901.29 total 50.04 last 82.16) total: 4500 open/close in 53.29 seconds: 84.44 ops/second - /unlink 1537 (time 1780385917.32 total 10.01 last 153.62) - /unlink 3204 (time 1780385927.32 total 20.01 last 166.70) total: 4500 /unlink in 27.44 seconds: 163.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 803 (time 1780385947.73 total 10.00 last 80.29) - open/close 1730 (time 1780385957.74 total 20.01 last 92.65) - open/close 2602 (time 1780385967.75 total 30.01 last 87.13) - open/close 3588 (time 1780385977.76 total 40.02 last 98.51) total: 4500 open/close in 48.76 seconds: 92.30 ops/second - /unlink 1904 (time 1780385999.20 total 10.00 last 190.37) - /unlink 3617 (time 1780386009.20 total 20.00 last 171.30) total: 4500 /unlink in 25.06 seconds: 179.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 948 (time 1780386027.00 total 10.01 last 94.73) - open/close 1963 (time 1780386037.00 total 20.01 last 101.46) - open/close 3080 (time 1780386047.01 total 30.02 last 111.63) - open/close 4053 (time 1780386057.01 total 40.02 last 97.30) total: 4500 open/close in 45.01 seconds: 99.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1698 (time 1780386074.71 total 10.00 last 169.77) - /unlink 3740 (time 1780386084.71 total 20.01 last 204.13) total: 4500 /unlink in 23.48 seconds: 191.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 1006 (time 1780386101.30 total 10.00 last 100.58) - open/close 2133 (time 1780386111.31 total 20.01 last 112.60) - open/close 3162 (time 1780386121.32 total 30.02 last 102.82) - open/close 4074 (time 1780386131.32 total 40.03 last 91.13) total: 4500 open/close in 44.11 seconds: 102.01 ops/second - /unlink 1740 (time 1780386147.92 total 10.00 last 173.92) - /unlink 3586 (time 1780386157.93 total 20.01 last 184.52) total: 4500 /unlink in 25.03 seconds: 179.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 963 (time 1780386175.79 total 10.01 last 96.23) - open/close 1932 (time 1780386185.80 total 20.01 last 96.87) - open/close 3005 (time 1780386195.80 total 30.01 last 107.26) - open/close 4002 (time 1780386205.80 total 40.02 last 99.69) total: 4500 open/close in 45.01 seconds: 99.99 ops/second - /unlink 2142 (time 1780386223.10 total 10.00 last 214.16) - /unlink 4298 (time 1780386233.10 total 20.00 last 215.59) total: 4500 /unlink in 20.95 seconds: 214.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 1268 (time 1780386246.16 total 10.00 last 126.77) - open/close 2383 (time 1780386256.16 total 20.00 last 111.48) - open/close 3572 (time 1780386266.17 total 30.02 last 118.76) total: 4500 open/close in 37.74 seconds: 119.23 ops/second - /unlink 2090 (time 1780386285.79 total 10.00 last 208.95) - /unlink 4057 (time 1780386295.80 total 20.01 last 196.64) total: 4500 /unlink in 21.93 seconds: 205.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 1317 (time 1780386309.77 total 10.00 last 131.67) - open/close 2635 (time 1780386319.78 total 20.01 last 131.73) - open/close 3870 (time 1780386329.78 total 30.01 last 123.49) total: 4500 open/close in 35.01 seconds: 128.53 ops/second - /unlink 2655 (time 1780386346.77 total 10.00 last 265.49) total: 4500 /unlink in 16.87 seconds: 266.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 1353 (time 1780386365.52 total 10.00 last 135.24) - open/close 2784 (time 1780386375.53 total 20.01 last 143.05) - open/close 4208 (time 1780386385.53 total 30.01 last 142.33) total: 4500 open/close in 32.61 seconds: 137.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 2541 (time 1780386400.11 total 10.00 last 254.03) total: 4500 /unlink in 19.19 seconds: 234.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 1274 (time 1780386421.76 total 10.00 last 127.35) - open/close 2429 (time 1780386431.77 total 20.01 last 115.47) - open/close 3465 (time 1780386441.77 total 30.01 last 103.57) total: 4500 open/close in 38.88 seconds: 115.75 ops/second - /unlink 2628 (time 1780386462.83 total 10.00 last 262.74) total: 4500 /unlink in 18.73 seconds: 240.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 1145 (time 1780386483.54 total 10.00 last 114.48) - open/close 2352 (time 1780386493.54 total 20.00 last 120.70) - open/close 3609 (time 1780386503.54 total 30.01 last 125.62) total: 4500 open/close in 37.67 seconds: 119.44 ops/second - /unlink 2172 (time 1780386523.78 total 10.00 last 217.13) - /unlink 4238 (time 1780386533.78 total 20.00 last 206.58) total: 4500 /unlink in 21.11 seconds: 213.13 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11196: 94008 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg455-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg455-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg455-server stop mds service on oleg455-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg455-server stop mds service on oleg455-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg455-server unloading modules via unload_modules_local on: 'oleg455-server' oleg455-server: oleg455-server.virtnet: executing unload_modules_local oleg455-server: modules unloaded.