NOTE -- I've added a new blog entry [1] detailing the steps necessary to run Fedora Core 2 (final) under VMware Workstation 4.5.2

After a bit of tweaking this evening, I got Fedora Core 2 Test 3 running under VMware 4.5.1. Here's how.

Host Configuration:

• Toshiba Satellite Laptop
• 512MB RAM
• 40GB Internal HD
• Windows XP Pro SP1 with all updates
• VMware 4.5.1

Guest VM Configuration:

• Fedora Core 2 Test 3 (downloaded .iso files)
• "Other Linux 2.6.x Kernel"
• 256MB RAM
• 4GB HD (/dev/sda1 = /boot, /dev/sda2 = /)
• 1GB HD (/dev/sdb1 = <swap>)

Installation

1. Boot installation CD.
2. At boot: prompt, type "linux vdso=0". This is very important, otherwise VMware will crash.
3. Install as usual, configure however you like. You'll need to ensure "vdso=0" is set in the "advanced boot configuration" options.
4. If you just reboot at the end of the install, the system will "freak out" a bit trying to start the x server. You can avoid this by editing the boot line (at the Fedora grub splash screen) and appending a 3 to the line.
5. Boot.
6. Unpack the VMware Tools tarball, then apply the patch below (vmware-fc2t3.patch) to vmwares-tools-distrib/bin/vmware-config-tools.pl.
7. Install VMware tools. Use the defaults for all questions, except say NO when asked if you want to compile a vmhgfs module. It doesn't seem compile with the 2.6 kernel headers.
8. Reboot, and enjoy.

Notes:

• Thanks to Daniel Arial at VMware for the "vdso=0" tip. I would have never found this on my own!
• Don't take this patch or these instructions too seriously. I just came up this this while hacking around tonight. All of the usual disclaimers apply. Your mileage may vary. I am not responsible for data loss, unusual hair growth, or any other rude behavior exhibited by your computer. You have been warned.
• In my experience, Fedora Core 2 Test 3 is extremely slow under VMware 4.5.1 (much slower than FC1). It takes about 11 minutes (!!) from the grub splash screen to the GUI login prompt. This is probably in large part due to the eleventy-bazillion services installed by default.
• I've noticed some bizarre problems with initialzing the guest keyboard and mouse. Apparently, if the VMware VM doesn't have focus while the guest OS is booting, there is a non-zero chance the keyboard will not initialize correctly. Or, the failure could just be completely random; I don't know yet. FWIW, I've never seen this with other guest OS's, only FC2T3.

---

vmware-fc2t3.patch:

--- vmware-config-tools.pl.before	2004-05-11 19:51:19.608817712 +0200
+++ vmware-config-tools.pl	2004-05-11 20:55:42.508567688 +0200
@@ -2788,7 +2788,7 @@
 }
 
 sub xserver4 {
-  return xserver_bin() . '/XFree86';
+  return xserver_bin() . '/Xorg';
 }
 
 sub xserver3 {
@@ -2984,7 +2984,7 @@
   my $sub;
 
   $xversionAll = direct_command(shell_string(xserver4()) . ' -version 2>&1') =~
-    /XFree86 Version (\d+\.\d+\.?\d*)/ ? $1: '0.0.0';
+    /XFree86 Version (\d+\.\d+\.?\d*)/ ? $1: '4.3.0';
 
   # This search order is issued from the XF86Config man page.
   if (defined $ENV{'XF86CONFIG'} && file_name_exist($xconfig_path . '/' .
@@ -2994,6 +2994,8 @@
            file_name_exist('/usr/X11R6/etc/X11/' . $ENV{'XF86CONFIG'})) {
     $xconfig_path = '/usr/X11R6/etc/X11';
     $xconfig_file_name = $ENV{'XF86CONFIG'};
+  } elsif (file_name_exist($xconfig_path . '/xorg.conf')) {
+    $xconfig_file_name = 'xorg.conf';
   } elsif (file_name_exist($xconfig_path . '/XF86Config-4')) {
     $xconfig_file_name = 'XF86Config-4';
   } elsif (file_name_exist($xconfig_path . '/XF86Config')) {
@@ -3146,6 +3148,18 @@
           $xversionAll);
 }
 
+sub fix_gpm_file {
+  my $gpm_file = '/etc/sysconfig/gpm';
+  if (file_name_exist($gpm_file)) {
+    my %p;
+    undef %p;
+    backup_file_to_restore($gpm_file, 'GPM_FILE');
+    print "Fixing " . $gpm_file . "\n";
+    $p{'^MOUSETYPE="exps2"$'} = 'MOUSETYPE="ms"';
+    internal_sed($gpm_file . $cBackupExtension, $gpm_file, 0, \%p);
+  }
+}
+
 sub fix_mouse_file {
   my $mouse_file = '/etc/sysconfig/mouse';
   #
@@ -3179,6 +3193,8 @@
       }
     }
   }
+  system('touch ' . $mouse_file);
+  fix_gpm_file();
   return $enableXImps2;
 }
 
@@ -3499,6 +3515,7 @@
     if ($major == 4 && $minor >= 2 &&
         file_name_exist($cX4MouseDriverFile)) {
       $line =~ s/%MOUSE_DRIVER%/vmmouse/g;
+      $line =~ s-/dev/mouse-/dev/input/mice-;
     } else {
       $line =~ s/%MOUSE_DRIVER%/mouse/g;
     }