Agavi 1.0.8 is now available for download at www.agavi.org, Composer and via the PEAR channel.
Apart from being the first stable release available via Composer, this release only contains some minor bugfixes.
As always, check the CHANGELOG for the full list of changes and fixes.
We’re thrilled to announce the immediate availability of Agavi on GitHub!
We realize it took much longer than anybody anticipated, but thanks to the combined hard work of Dominik, David and Noah, and with the invaluable assistance of Steffen, Thorsten, Tom and everybody else who provided their support, we finally did it!
A big thumbs up goes out to everyone out there for sticking around during that time and not abandoning, what we think, is still one of the best PHP frameworks available today.
There are some immediate effects of the migration:
While this already is a major step forward, there is still more work to do to bring Agavi back up to speed, starting with small improvements to the surrounding infrastructure (API docs generator update, port documentation to Sphinx, etc).
Basic details regarding GitHub, package downloads and installation instructions are already on the website; over the next few weeks, we will work on updating other parts of the website, the build and release tool chain, and any other components that could benefit from the new possibilities that GitHub offers.
We will start the 1.1 release cycle in 1-2 months from now. The current changelog is
available at https://github.com/agavi/agavi/blob/master/CHANGELOG and we should be
able to bring even more goodies into the release, including a much better
authorization and ACL support, and support for ext/intl in the i18n module.
After 1.1 there will be at least one more minor release. Please check https://github.com/agavi/agavi/issues?milestone=22&state=open for list of planned features. The last 1.x release will be a long term support (LTS) release, which we plan to support for at least 18 months. The plan for 2.0 is not yet finalized but it will be a mostly incremental update with some necessary breaking changes. This includes moving everything to namespaces and cleanup of a few unsound spots in the code.
Right now we haven’t completely figured out the new development model for branching and pull requests for permanent contributors.
Until we figured the best development model, we invite everyone to start creating pull requests for master! We will then make sure it ends up in the next bugfix or minor release, whichever is applicable.
For us, the move to GitHub is just the beginning – we have many plans and ideas to advance Agavi in the future. But we cannot do that without you.
Happy forking!
Assuming you use the fabulous svn2git to convert your SVN repository to a Git one, you might still run into a problem: merge history.
If you have always used SVN 1.5 or newer and thus have svn:mergeinfo properties for all your merges, you can ignore this part, as git-svn will correctly identify merges from, say, a branch back to trunk (or master once the conversion is done) and attach two parents to that commit: the previous commit in master, and the last commit from the branch that’s being merged.
In such a case, your history will look something like this:
$ git log --pretty=format:\"%h %ad | %s%d [%an]\" --graph --date=short
* 0d34dbe 2009-08-12 | allow passing of arbitrary arguments to ant (HEAD, master) [felix]
* a199e71 2009-07-06 | merging the changes from [3853:4186/branches/felix-tutorial-app-rewrite] major rewrite of tutorial apps major rewrite of build scripts [felix]
|\
* | 2121341 2009-07-06 | removing staging apps for backmerge, somehow svn mucks up if I don't [felix]
| * 145e1aa 2009-07-06 | merging the changes from [4165:4184/branches/felix-build-scripts-rewrite] fixes #1115 (felix-tutorial-app-rewrite) [felix]
| |\
| | * 2a9b5cc 2009-07-06 | fix excludes for svn directories (felix-build-scripts-rewrite) [felix]
| | * ec374ec 2009-07-02 | reorganized the stylesheets a little, make a custom base that does not set doctypes on the output and a stylesheet for the website as well as for the xhtml documentation that inherit from it [felix]
| | * f119381 2009-07-01 | modified the toolkits build system to allow overriding the xsl for the toc-generation [felix]
| | * 576b71b 2009-06-29 | creating branch for build scripts rewrite, fixing build problems in dita-ot 1.4.2 and greater refs #1115 [felix]
| |/
| * 6e8e682 2009-06-26 | added a note about Action::executeRead() to the chapter about module creation fixes #1091 [felix]
| * 976c384 2009-02-18 | creating a branch for the rewrite of the tutorial apps to incorporate latest best practices. [felix]
* | 2121341 2009-07-06 | removing staging apps for backmerge, somehow svn mucks up if I don't [felix]
|/
* fd87653 2008-10-31 | dump current versions of refguide [mikeseth]
If, however, you did not have merge tracking yet, the history will look like this:
$ git log --pretty=format:\"%h %ad | %s%d [%an]\" --graph --date=short
* 0d34dbe 2009-08-12 | allow passing of arbitrary arguments to ant (HEAD, master) [felix]
* a199e71 2009-07-06 | merging the changes from [3853:4186/branches/felix-tutorial-app-rewrite] major rewrite of tutorial apps major rewrite of build scripts [felix]
* 2121341 2009-07-06 | removing staging apps for backmerge, somehow svn mucks up if I don't [felix]
* fd87653 2008-10-31 | dump current versions of refguide [mikeseth]
The reason becomes apparent when we use git-rev-list:
$ git rev-list --all --graph --oneline --date-order
* 0d34dbe allow passing of arbitrary arguments to ant
* a199e71 merging the changes from [3853:4186/branches/felix-tutorial-app-rewrite] major rewrite of tutorial apps major rewrite of build scripts
* 2121341 removing staging apps for backmerge, somehow svn mucks up if I don't
| * 145e1aa merging the changes from [4165:4184/branches/felix-build-scripts-rewrite] fixes #1115
| | * 2a9b5cc fix excludes for svn directories
| | * ec374ec reorganized the stylesheets a little, make a custom base that does not set doctypes on the output and a stylesheet for the website as well as for the xhtml documentation that inherit from it
| | * f119381 modified the toolkits build system to allow overriding the xsl for the toc-generation
| | * 576b71b creating branch for build scripts rewrite, fixing build problems in dita-ot 1.4.2 and greater refs #1115
| |/
| * 6e8e682 added a note about Action::executeRead() to the chapter about module creation fixes #1091
| * 976c384 creating a branch for the rewrite of the tutorial apps to incorporate latest best practices.
|/
* fd87653 dump current versions of refguide
The merge commits (145e1aa and a199e71) each have only one parent and do not point to the merged commits. This ancestry is then lost when running git log. As a result, it will later be really difficult or impossible to find the commits from those branches again, especially if the corresponding branches have been removed from the origin.
You can override parents of a commit in Git using a file called grafts, which needs to reside in .git/info/. In consists of several lines with commit hashes separated by a space character, where the first hash is the commit you want to define parents for, and the following hashes are the parents.
Note: you must use full hashes, not abbreviated ones.
To fix the history above, we must define the parents in .git/info/grafts:
a199e7194c5a4382c7d1057c84a55fcb535c8ed2 2121341005ef4fee0c1755d9ee083b6f0fbaf295 145e1aa687c7b1277c683b8116324a1552ba6454
145e1aa687c7b1277c683b8116324a1552ba6454 6e8e6828f7be352571faccef981568f4acc9ccb1 2a9b5ccf4b0870973508b37bef8d7759889a4f27
Note: for a merge from, say, master to a branch (to sync the branch with master/trunk), you would use $synccommit $lastmastercommit $lastbranchcommit.
If we now run git log --pretty=format:\"%h %ad | %s%d [%an]\" --graph --date=short, things will look just like we expected.
We can then run git filter-branch to make the grafts permanent (see step “Manual Conversion” below). Note that this will change commit hashes.
svn2git http://svn.example.org/repos --no-minimize-url --metadata --authors authors.txt.git/info/grafts (you can use git svn find-rev r$rev $branch to find a Git commit hash for an SVN revision number)git filter-branch --tag-name-filter cat -- --all to make the grafts permanentgit for-each-ref --format="%(refname)" refs/original/ | xargs -n 1 git update-ref -d to clean up the backups created by git-filter-branch.git/info/graftsFor the Agavi migration, we wrote a script to automate this process; it uses a grafts file that contains SVN revision numbers (and their branch names) and converts it to a grafts file fit for consumption by Git. We will put that script up on GitHub together with the rest of our migration toolkit in a few days.
The Agavi documentation repository contains a broken branch, where the branch was made from only a portion of trunk, not from the whole of trunk, in revision 4285. When running this through svn2git, the whole history would be terribly confused and a lot of commits would appear as duplicates since git-svn interpreted the branch point wrong.
The fix in this case was to simply ignore that branch and only use its merge back to trunk (in revision 4308), as the loss of history information in this case was negligible. We told svn2git top stop at the revision before the branch was created, and resume at the revision where the branch gets merged back to trunk (this required a patch to svn2git):
$ svn2git http://svn.agavi.org/documentation --revision 2721:4284 --no-minimize-url --metadata --authors authors.txt
$ git svn fetch -q --revision 4308:HEAD
$ git merge -q svn/trunk
We’re only merging svn/trunk in this case as all the commits from 4308 onwards were made to trunk. Also, between 4284 and 4308, all commits in this repository were in the branch we’re skipping - had there been commits to trunk or other branches, things would have been a tiny bit more complicated, with several calls to git svn fetch and git merge to put the pieces together correctly.
Sometimes, you commit a change, but then you realize that you did not mention the ticket ID in your commit message. Or, worse: you use the wrong ticket ID.
The usual remedy is adding this info to the ticket by hand. On the Agavi Trac, when a ticket ID was incorrect in a commit message, we would typically remove the corresponding entry from the wrong ticket and move it to the correct one.
When migrating to GitHub and replaying all commits and ticket changes, that becomes a problem of course: the commit message still contains text like “fixes #123” when in fact it was supposed to reference ticket #124.
It is therefore necessary to programmatically find and mitigate such problem cases and addressing them before importing things on GitHub. Failure to correct just a single of these problems may break the whole import by having tickets remain open randomly even though they were supposed to be closed and so forth. Here is a list of situations that need to be addressed:
Sometimes a reference to a ticket ID in a commit message does not show up in the corresponding ticket. There are several reasons why this may happen:
Aside from the first point above, there are other subtle but important differences between how Trac and GitHub parse commit messages:
People do stupid things in their ticket database, so aside from the situation we already discussed where a commit referenced the wrong ticket number, we’ll have to watch out for:
Finally, a few things in commit messages or tickets may not be a problem for the import, but could still be nice to fix:
svnmerge have the nasty habit of repeating all merged commit messages and thus cluttering the history.
Before modifying the ticket and comment contents themselves (to convert them from TracWiki syntax to Github Flavored Markdown), a few more fundamental changes are necessary:
That one is fairly easy:
$ sqlite3 trac.db 'SELECT id FROM ticket;' | php -r '
$ids = file("php://stdin", FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES);
echo implode("\n", array_diff(range(1, max($ids)), $ids)) . "\n";
'
The resulting list needs to be stored somewhere for later, when we replay all ticket changes - that script will produce empty dummy tickets for the missing IDs to make sure all ticket IDs line up between Trac and GitHub.
We only need SQL for this:
sqlite3 trac.db '
SELECT DISTINCT (user || " = ") FROM (
SELECT reporter AS user FROM ticket
UNION ALL
SELECT author AS user FROM ticket_change
UNION ALL
SELECT author AS user FROM attachment
) ORDER BY LOWER(user) ASC;
'
The result can be put into a file that is then filled by hand with references to people. In our case, we chose to use either a GitHub username (if known, requires some research in certain cases), half of the person’s e-mail address, or a nickname in quotes:
anonymous = "anonymous"
david = @dzuelke
steve@REDACTED.com = steve@…
“REDACTED.com” is obviously not what was really in there; the ellipsis (“…”) is correct however.
This map will be used by a script later when replaying ticket changes via the GitHub API in order to embed information about the original authors. By using GitHub usernames with the “@someone” notation, a link will be created; we just have to get things right on the first attempt or some people will complain about notification spam ;)
Very straightforward as the GitHub issue tracker supports milestones. They existing milestones in the Trac database are simply ported over to GitHub; milestone “AgaviForge” will be dropped however (and the corresponding tickets later reassigned).
Versions as registered in Trac will simply be imported as labels like “0.10.2” or “1.0.8”. For now, we will import all versions except for “HEAD”, “HEAD-1.0” and “HEAD-0.11” since those only cause clutter and thus far have rarely been used.
This will create a considerable amount of labels just for the versions; we might eventually merge them all into series labels (like “1.0.x” or “1.1-latest”, similar to e.g. Ruby on Rails). When doing that, we might also drop old series labels that are not maintained anymore, simply to cut down on clutter.
One last thing to decide upon of course is the color codes for the versions. No great ideas yet, as the colors are shared with components and ticket types (see below).
Components will be migrated to labels with identical names, with the exception of “documentation” (will be moved to a separate repository), “website” (never has been part of the SVN repos, but will be moved to its own GitHub repository eventually) and “OTHER” (no label will be assigned to such issues).
Again, color coding is something that’s left to be decided (suggestions welcome).
There currently are just three issue types: “defect”, “enhancement” and “task”. Those will be migrated as-is, with defects getting a flashy red color, enhancements a nice green, and tasks… well… turquoise maybe?
Priorities will simply be dropped, they have never proven to be useful. Same for severities, although we might have a dedicated label for blockers, indicating that resolving such an issue may not be postponed until a later release.
Some of the tickets in the Agavi Trac have attachments; these attachment files need special treatment as the GitHub issue tracker itself has no support for attachments.
We’ve created a repository (agavi/trac-ticket-attachments) for this purpose and simply imported the everything from trac/attachments/tickets, which contains just a bunch of folders with the ticket ID and the attachments of the ticket inside them. Some of the files have %20 in them as the uploaded files had spaces in their names; we changed those to actual space characters so the URL escape does not end up in the file name in the Git repository.
We will later rewrite the ticket history so the attachment links point to the corresponding files in this repository.
One thing we’ve always put a focus on is having tickets for every change we make, and using these tickets extensively in commit messages and from other tickets so a quick svn blame on a line not only shows a description of the change, but also gives a ticket number that can be looked up on Trac to follow a discussion of the change.
On numerous occasions, I looked at a bit of code, found it to be incredibly stupid, but then did an svn blame anyway to figure out when the change was done. Looking up the ticket on Trac then usually gives a list of all the commits for that change, and maybe even related tickets and discussions. Often enough, that line of code then turned out to not be so stupid after all, and the day was saved.
Good examples are tickets #502 or #1199, which link to related and duplicate tickets, have discussions in them and of course commits references.
All these tickets contain valuable information, so it is vital to preserve all this history. However, the GitHub ticket API does not allow a timestamp in updates, which means that any ticket and any comment will have the date and time of import, and not the old date and time. Commits from the repository, however, will contain the correct timestamps as git2svn preserves that information.
We tested how the GitHub issue tracker behaves with respect to timestamps, and found the following:
The result is that we can now build a database of commits and ticket changes that are in the order of their timestamps, and replay them step by step, meaning that we will convert the repos, then push a commit, then replay a ticket update or creation, then push another commit and so forth. The result will be that a ticket might contain a comment (with a timestamp of, say, January 2 2012 13:04:12 because that’s the time when our API client replayed that change) followed by a commit reference that closed the ticket (with a timestamp of, say, August 4 2007 08:09:12, because that’s when that change was committed).
Ultimately, this means that we will be able to use GitHub’s native support for “refs #123” or “closes #456”, which will look much nicer than manually having to construct comments through the API about commits referencing the issue.
The remaining issue of course is that any comment and ticket will have the wrong date and time, so we’ll have to include that in the next (along with a notice saying that the ticket/comment was migrated from the old Trac, just so people know), but it simply seems like there is no way around that with the GitHub API.
Regarding severities and priorities, we might drop these altogether along with keywords; components can be done via labels, and so can ticket types (enhancement/task/defect) and versions. Milestones are supported natively by the GitHub issue tracker, so they won’t be a problem.
Certain changes in GitHub, such as changing milestones, labels or assignees, are not displayed in the web interface, even though the changes appear to be recorded internally and are available through the API, so the information itself is not lost.
We have just begun working on migrating trac.agavi.org and svn.agavi.org to GitHub; there is whole range of things to consider, from referencing users correctly over rewriting commit messages so the correct Git commit references are included to preserving the history of tickets in their right order.
We will add more blog posts as we move along to document the process in case other projects would like to do the same, and also make our migration scripts available if they are sufficiently generic to be useful to others.
Agavi 1.0.7 is now available for download at www.agavi.org and through the PEAR channel.
The release contains just two minor changes over RC2, so I’ll simply quote from the final release notes:
The getCredentials() method on an Action is no longer called unconditionally (i.e. whether or not the isSecure() method returned false) but only if the Action is “secure”.
It is now possible to manually call shutdown() on any database adapter to close the underlying connection; another call to getConnection() will cause a reconnect.
AgaviFormPopulationFilter will now populate multiple forms in the order specified in the “populate” request attribute (when populating via an array with form IDs as keys and parameter holders as values) in namespace “org.agavi.filter.FormPopulationFilter”); however, if the value for a key in the “populate” array is boolean true (to re-populate from request data), this form will always be handled first, so error messages are inserted on that form first.
Testing is now compatible with both PHPUnit 3.5 and 3.6. If you want to specify code coverage filters, AgaviTesting::getCodeCoverageFilter() returns the correct instance (singleton or not depending on the PHPUnit version) for you to use. A base constraint class named “AgaviBaseConstraintBecausePhpunitSucksAtBackwardsCompatibility” can be used for constraints that work with both PHPUnit 3.5 and 3.6; if you implement the new matches() method introduced in PHPUnit 3.6 instead of the old evaluate(), it will automatically be called in the proper fashion depending on the PHPUnit version.
AgaviTesting::dispatch() can now call exit() with the appropriate shell status code (the same as returned by a vanilla PHPUnit run) to indicate success or failures/errors to the calling process. This behavior is triggered when the new optional second argument, defaulting to false, is set to true. If set to false, it returns the PHPUnit result object that may be used by custom code to perform further analysis of the test run.
Several other minor changes and fixes are included in this release as well; most notably, AgaviBooleanValidator’s casting and exporting logic has been repaired, and AgaviConsoleRequest now properly creates an AgaviUploadedFile object with STDIN contents (when configured to read those) instead of a plain array. The PHPTAL renderer now supports configuration of character encoding via parameter “encoding”.
The timezone database was updated to version 2011n.
As always, check the CHANGELOG for the full list of enhancements, changes and fixes.
Agavi 1.0.7 RC2 is now available for download at www.agavi.org and through the PEAR channel.
A few minor fixes and changes have made it in since RC1; most notably, PHPUnit 3.6 should work fine now, and the Form Population Filter, when populating several forms (via an array as the value for “populate” with form IDs as keys) now gives precedence to boolean true values (so error messages are inserted on re-populated forms first) and processes forms in the order given in the “populate” array rather than in the order they appear in the document.
Please test this thoroughly and give feedback if necessary so we can roll a final version before the holidays.
© 2005–2011 The Agavi Project. Site design by QUERIDODESIGN, original logo by Dariusz Zieliński.