I think Facebook might be better for getting members if you upload a lot of videos, pictures, articles, links and news about your forum on your Forum's Facebook group. You might get some more visitors to your website. But, you need to promote your Facebook group a lot to get people to check your FB group out, Like, and click on a link to visit your forum.
But, Google Search is easier to get traffic to your forum, but fewer people join forums which they find on Google unless the forum is very good, and active. I think most people from Google only look for information, and visit another site compared to social networks which are usually more social.
Most of my current members come from post exchanges, or from clicking my signature link on other forums. I recognize their avatar, and screen names from other forums on my forum.
I would say definitely Google from personal experience. Facebook is good, but I see it more as a platform of communication between people that are already members on the forum rather than getting new members.
I'd say Google to be honest.. FB I think will be the death of forums eventually.. especially for those that are still struggling to establishing themselves.
But, it is getting a lot harder to rank high in the first and second page in Google these days for important keywords like "Video game website" or "gaming" since the top spots will be taken by huge websites like IGN, Gametrailers, Gamespot, GameSpy, and other huge gaming sites with a forum for members.
Plus, the spammers and richer website owners are always finding ways to rank high in Google by making thousands of backlinks to their websites by using spambots or paid links, and other methods to optimize their site for Google.
With Facebook, you can convince your lists of online friends, family members, co-workers, members of your community, and people you know online and offline by sending them messages on Facebook, or posting on their wall to join your forum. If you have a huge friendlist in the hundreds, you can get a lot of members if they click on your link.
With a Facebook group, you can also customize it to convince people to join your forum by posting pictures on your forum, and videos about your forum. Facebook Groups also rank high on search engines.
Plus, if you know how to make Facebook Apps like Mafia Wars, Cityville, Farmville, etc you can get more members from people who use and promote your apps. Zynga the huge Facebook gaming apps developer would probably be just another flash game sites if they just relied on Google, and did not post their apps to Facebook.
