Create a custom menu item with the following settings:
URL: #LoginPopup
Label: Sign in
CSS Classes: popup -function-is-user-logged-in
Click here to see an example.
There may be a better solution, but here is what I have found to work regarding the BuddyBar Log in link displaying the popup window – using your FTP program, sign in to your root Wordpress install and navigate to wp-content/plugins/buddypress/bp-core. Download a copy of the “bp-core-buddybar.php” file to your computer, to err on the side of caution, rename the file bp-core-buddybar.php on your server to orig-bp-core-buddybar.php, open(with a plain text editor, dreamweaver, coda, etc.) the buddybar file you downloaded, go to line 415, modify as shown below, upload modified version and test it out. If something goes wrong or you get an error, simply delete the file you just uploaded and rename “orig-bp-core-buddybar.php” to “bp-core-buddybar.php”
Original code:
echo '<li class="bp-login no-arrow"><a href="' . bp_get_root_domain() . '/wp-login.php?redirect_to=' . urlencode( bp_get_root_domain() ) . '">' . __( 'Log In', 'buddypress' ) . '</a></li>';
Click here for screenshot of original code
Change to something like this to show popup:
echo '<li class="popup bp-login no-arrow"><a href="#LoginPopup">' . __( 'Log In', 'buddypress' ) . '</a></li>';
Click here for screenshot of modified code
Please keep in mind that editing the core files of wordpress and/or buddypress is never a “good” idea – and ALWAYS, backup your DB and site files before performing such modifications. Please use good judgement and modify core BP files at your own risk.
Hope this helps!