How to Orient the Local Rotation Axis (LRA) of a Joint
If you're new to what a local rotation axis is, let's start with a scenario you may have already seen and didn't know how to fix.
Let's say I'm creating a new rig for a character and working on the arm specifically. I click out a series of joints for the arm, but they aren't quite in the right place for my mesh so I have to move them to get them in the position of the arm. (Unless you built your arms straight out along the grid, you usually have to move them some. Like if you built them in the front view, you'll likely have to move them to line them up with the arm in the side view. And for the sake of this tutorial, let's assume I'm really bad at placing joints so it's easier to see the point I'm trying to demonstrate.) (Fig.01).
After clicking out the joints I'm going to move them into the place I want them. Don't ever rotate your joints into place if you can avoid it. You always want to have clean rotations on your joints so your animators can zero out the rotations later if needed to get the rig back into its bind pose. If you do need to rotate them to get them into position, you should use Modify > Freeze Transformations to reset your rotations (Fig.02).
TIP: If you want to move a joint at the top of a chain without moving its children, you can just move its pivot point. Because that's all joints are: pivot points. You can hit the Insert key to toggle Pivot Mode on and off. You can also hold down the D key to move it as long as you are holding down the key (Fig.03).
The problem we have now is if I want to rotate the elbow. From an animator's stand point, since the elbow only bends on one axis, it'd be easiest to animate the elbow if it also used on animation axis to bend. But if we look at it right now, it's going to bend on multiple axes in the channel controls, to appear to be bending in on the axis (e.g: rather than being able to just rotate Rotate X, you have this crazy combination of Rotate X, Y and Z, which is impossible to get a real handle on in your graph editor later) (Fig.03).
But this is an easy fix. We can orient what is called the Local Rotation Axis. You may have noticed that if you just click out a series of joints and don't move them after creating them, there is one axis that is lined up with the joints and it probably makes more sense when rotating it. There's a good chance that if I had built the chain in this position to start with, it'd be oriented correctly and we wouldn't have that issue. But that's rarely the case when placing joints.
To edit the LRA of the joint we want to start off with having the joint we want to edit, or its chain's parent selected. We're going to go into Component mode and then turn on the filter for Miscellaneous Components (The ? button.) (Fig.04 - Fig.05).
In this mode we can now see our LRAs. We can edit them by simply rotating them. But first, I want to get a bit more technical about how they're aligned. I want to be able to set up the joints where at least one axis can spin the bone like a top. That means one of those axes has to be aligned right up with the joint. How do we do that? We have it aim at its child joint. But easier than that, there is actually an Orient Joint command that does it. For a selection, we either need to select each LRA we want to orient, or I can just go back to Object Mode and select the base of the chain I want to orient. In this case I will select the chain on the arm. Then I will go to Skeleton > Orient Joint (Fig.06).
This will aim the X axis of the joint to point towards the next child joint below it in the chain. This will allow me to have the proper spin or twist axis to this joint. As well, if you wanted a different axis to be aimed, you can change which axis will be oriented in the options (Fig.07).
When we look at it now, this is a much better alignment. I can also bend it in just one axis now to bend the elbow. But perhaps this isn't quite lined up with the crook of the elbow; in this case, if I want to spin it some to line it up better, I can rotate it on its X axis (we just rotate X now at this point, because it's the one axis we've aligned so far) until the Y axis feels like it's in the proper spot to bend the elbow. This part I just eyeballed to get right (Fig.08).
It's a slight nudge but it'll make all the difference in how my elbow bends and, when it does, if the hand ends up over the shoulder or not. To test it out, I go back into Object mode and bend the arm to see if it lines up (Fig.09).
And that's what I want to see. I'm bending in only the one axis, Rotate Y and it lines up over the shoulder, like the bones should.
The elbow is just one example. I'd do this on the knee too, but any other joint as well. The spine here is another good example. In the Orient command, it tries to pick an axis to aim up. Well, if you build your spin in an s-curve, up depends which side the curve is on (Fig.10).
We are orienting our LRAs because we want our rigs to be logical to work with. For example, if I grab any joint in the spine, I probably want something like a positive number rotation to rotate the rig forward and a negative number rotation to be backwards. (It doesn't match which goes forward and which goes backwards. Just that it's consistent from joint to joint.) Let's look what happens right now if I select each joint in my spine right now and rotate it +50 degrees (Fig.11).
Everything bends backwards apart from that third joint up in the chain. Why? Well if you look at the previous image the Y axis is facing the opposite way from all the other Ys. To have one joint that bends the opposite direction from all the other joints would be confusing to animate with. So we're going to grab that LRA and spin it 180 degrees to match the others.
The Rotation tool does have a snap feature. So I'm going to double click the Rotation tool in the toolbar to bring up its options panel. Under Rotate Settings, I'm going to check Discrete Rotate and set the step size to 90 (90 degrees is easier than trying to snap to a full 180). Then I will go into Component mode again, select that rogue LRA and spin it in X 180 degrees to match the others (Fig.12).
This is what my spine should look like now (Fig.13).
And now it should bend like this. (Don't forget to turn off Discrete Rotate afterward to test with, unless you want to keep snapping to 90 degrees) (Fig.14).
Now this makes sense to work with. When I grab all the joints and rotate them in the same axis, they now all bend the same direction. And you would do this for any chain. The leg is another good example of where this can happen. For any chain, you typically want one axis to face towards the child joint, one to face "up" and one to face to the "side" and for all the LRAs in that chain to consistently face the way.
In conclusion, orienting your LRAs is all about logical animation in the end. If something is a hinge joint, it should be able to bend in just one axis to achieve that motion. Now there are motions that will still influence all three rotation axes and won't make as much sense here and there, but that has more to do with gimbal rotation order, which is a whole other issue, but orienting your LRAs is the first step to having any control over this issue. At this stage the point is to make sure things are easy to move and make sure things make sense in the way they move.
Now you don't always have to use the Orient Joint command. Sometimes you can go straight to just eyeballing a joint into place. Like with the jaw, I often just manually rotate the LRA into a position that makes sense for the Rotation tool. Since it generally only bends in one axis, I just try to make sure it stays lined up across the center (like down the middle of the model, its symmetrical still).
Keep your chains separate when using the Orient Joint command. You may notice here that my arm is parented to my spine. Well, if I try to Orient my spine with the command, that one joint there has two children. Maya will usually pick just one to aim to, but you don't know which one. Keeping your chains separate until after you orient your LRAs will make things simpler.
The Orient Joint command can only orient joints with a child joint to aim to. So what if it's an end joint that needs to be oriented? Often I build an extra joint on the end of my chains that I use just for orienting. After what I've built as the intended end of my chain has been oriented, I delete the extra joint I made. Sometimes joints are just there to have something to aim at for a moment.
You may notice that when you create joints, it auto-orients your LRAs on the first placement click. As long as you hold down your mouse button and don't let go, you can still position the joint where you want. The first time you let go though it will auto orient your LRAs for you. There are times where you can use this to your advantage and where it may make your work a little easier. After you reposition the joint after that, the LRA will keep facing the same direction it was.
Always do this from the start. Building on top of your mistakes rarely turns out well. You may have noticed that right now we can rotate the LRAs of the joint, and the actual joint itself doesn't gain any rotation values and stays zeroed out. If you tried to do this down the production line though, after it's skinned, you may see rotation values added or your skin may twist with your LRAs. Oriented LRAs are the good foundation of a rig you build upon, so make sure they're right to start with or you may end up having to redo work.