There are so many guilds out there that there will always be a "better and bigger" guild to join. Most guilds are just a bunch of people together who share the same cape. Ya know, the "Yay, I spent 2k for a cape -- join my guild!" kinda guilds. Your guild needs to have something unique that you offer, something that members see and think, "This guild offers something other guilds don't." They also want to see that there's activity, something exciting going on that's worth sticking around for. Solid leadership is also a key componet -- someone who's leading the guild in a direction that people want to go in.
Overall, though, building a solid guild takes time and committment. Our guild started back in betas and has grown to almost 40 members after many people coming and going (and being kicked

). We're selective in who we take as a member, too. We have never spammed chat channels in-game begging for people to join. Rather, we present something exciting to the GW community and wait for people who want to rally alongside us.
As a guild leader, don't get discouraged if people leave your guild. Personally, I feel that if someone isn't committed to the guild and doesn't want to be in it, then I'm glad that they leave. I don't want people in our guild who don't like it but stay just out of obligation. That's not very helpful for morale.
IMO, building a solid guild seems to be a long-haul process. Maybe for every 5 players you accept only 1 of them will be committed and stay. Yeah, it's discouraging to see the other 4 come and go, but don't let that make you neglect the 1 who stays. Stay positive about that 1 person instead of getting upset about the 4 who leave. Say, for example, you accept 40 members and see 32 take off (theoretically) but 8 will stay. That ratio seems to be a little high, but you get the point. With some time the guild will build. It just takes a while.
It also makes a lot more work for the leader and officers in recruiting. Fortunately, in our case, all applicants are ones that have approached us for membership instead of us actively trying to find people to join. That cuts down on a lot of time spent, too.
But anyway, as the leader, decide for yourself where your guild is going and find people who want to go in that same direction. A leader is not a leader if he's not leading anywhere. Leadership is much more than just having your name at the top of a guild list.