Cloning the Macs

Don’t think that I am a proponent of making the Macs on our network multiply, but rather making those on our network look the same… One of the keys to our AD and Mac integration was reducing the time it takes to deploy a mac on our network.  Earlier this year when I had to reinstall a mac book pro it took over 6 hours to install the base software and drivers (items that every user gets) in addition to installing the components that each specific user needs.  Knowing this was taking way too long I was on a quest to make this less painful.

Enter Apple Server’s System Image Utility… The Image utility allows for you to create a base system, prepare it for deployment over the network and distribute it to similar clients (Intel or PPC).  There are several options for creating the image 1. Pull the image from DVD or 2. Clone an existing machine.  The benefits of creating the image from DVD are a clean from factory default installation that can deploy fairly quickly over the network… We however we elected to clone an existing machine.  The clone allows us to add all the base software and drivers and then push that image with the base software already installed to another machine.

How its done.

• Install from DVD on to a machine that is the same vintage of processor as the machines that you plan to deploy the image to.
• In our case we have both PPC and Intel so we started making an image of each on two separate machines. 
• After the OSX installer is complete update the OS from Apple Updates add the software you want included and the preferences you would like configured. 
• In our case our base install includes: Mac Office 2008, Canon PS and UFR II Drivers, Disabling the onboard Bluetooth, Disabling the .DS_Store for network volumes, Disable Guest Account Access. (DO NOT INSTALL SYMANTEC BEFORE YOU IMAGE THE MACHINE… for some reason this causes the image to fail)
• After the base software and drivers are configured, go to Disk Utility and run permissions repair.
• Capturing the image can only be done on a secondary volume from where you are installing the OS.
• If the install is on a primary volume, you will have to boot the device in target mode from the Startup Disk System Preferences.
• If it is installed on a secondary volume you can boot to the primary volume to capture the image.
  
• On a machine with the OS X 10.5.3 Server Admin Tools installed (downloaded from  http://www.apple.com/support/downloads/serveradmintools1053.html or off the OS 10.5.3 Server Disk) Start the System Image Utility and it should find the volume you just created and updated.
  
• Select the Volume you want to image and choose netinstall and select customize.
• Add Enable Automated Installation and Create Image to your workflow then configure where you want to save the output files, select an index for each image you will create and choose RUN.
• After an hour or so the location you selected will have a folder/file ending in .nbi

How to Deploy the image: (Our configuration)

• Enable the NetBoot service in the Server Admin Console
• Next Configure the NetBoot Service by going to the settings
• On the General Tab Enable which device you want Netboot to run on (Ethernet)
• On the General Tab Select where you want to store the Images (Volume 2 for both Images and Client Data)
  
• Copy the .nbi to the location where you told the Netboot service to save the data.  /NetbootServiceLocation/Library/NetBoot/NetBootSP0
• Next Configure the Images on the Server Admin>Netboot>Settings> Images Tab
• Enable the Image you would like to NetInstall from.
• Select the Architecture that you would like to use this volume.
• Restart the NetBoot Service
  

How to boot the Machine and install the image:

• While booting the device press the ‘n’ key or select the network Network Volume in System Preferences>Startup Disk
  
• When the device boots a little world will display and then the machine will indicate that it is recovering a system image.

After the Image is restored, the machine will rename itself and add a digit to the end, so you can install this on as many machines at the same time and not worry about the issues you might have without running Sysprep on a Windows machine.  Simply rename the machine in System Preferences> Sharing and change the name and the local hostname.

Customizing the Sonicwall Content Filter

Tweaking our Sonicwall Gear Continues, and I have to say it responds quite well to our ever changing needs.  Several of our ministries needed our content filtering tweaked for their needs but IT and our Leadership team didn’t want to open entire categories in the filter to allow specific sites across the board especially since are using that content filtering for our core network AND our public Wifi, so enter Sonicwall Single Sign On Agent (SSO).

Sonicwall has offered the SSO agent since the release of version 4 last fall, but it took us quite a while to get this project on the top of the list and get it tested before we deployed it into production.  Basically the SSO agent allows for the user’s Windows workstation to authenticate the user who is logged into that workstation against the firewall.

The configuration process is fairly basic (although because of several issues I won’t say easy). 

What you need to make this happen:

• A Sonicwall Firewall with and active Content Filtering Service subscription and the CFS enabled
• The Sonicwall SSO agent installed on a networked machine (in our case a virtual server)
• A working MS Active Directory structure for your domain.
• User(s) added to groups defined in Active Directory.

The biggest hurdle we had was the SSO agent failing on us.  We would get everything running in our test environment and then the agent service would stop and then there was no web access.  You can set the device to fail open if the SSO agent fails, but we elected to drop down to the default CSF policy for all users if the agent fails.  After we downloaded a more recent version of the SSO agent, moved it to a more stable server and that seemed to resolve that issue.

First you need to adjust the content filter to allow access to the specific sites you want to allow some users to access. 

• The catch, you can’t allow/deny access to a specific url based on the logged-in user or group. To resolve this issue we had to first allow the specific sites we need to access through the content filter. This is a simple white list in the content filter. 
  • Note this excludes the url from all content filters
• Then utilize the firewall, which has the AD group specific granularity you need.
  • Create a firewall rule to allow access to the sites for specified groups and deny for all others.
  • This is all of course only on the authenticated side. Other zones and non Windows machines follow the default rules and provisioned the exceptions by user. Since firewall rules are higher priority than CFS the site can be an exception in the CFS but only those who are in the allow group are able to view the site.

Kudos to ACS OnDemand

During our recent outage Dean Lisenby and the Team at ACS came thru for us again.

I contacted Dean after hours to alert him to our outage and asked how ACS could help us have access to our data quickly in a read only format.  I quickly received a text message that he would contact me after his kids went to bed… (Family First for sure!) I received another text message about 25 minutes later asking which back we wanted to use and then another text message 10 minutes later saying to check my email.

The email gave me 5 user names and passwords to access our data via the ACS OnDemand system.  Dean had recovered our latest backup and we were able to access the data.

We were able to communicate to our staff which users had access and how to use the system by the next morning.

We elected to just use this installation as a reference, since our plan was to bring the production database and server back online as soon as the hardware was repaired.

Even though it might have appeared as minor to bring us online for just a day with data from a backup that we weren’t going to keep ACS made this a priority.

Thanks ACS!  And Thanks to Stacy Kennedy; Mark Williams; Darlene Winborne; Sylvia Watts and Dean Lisenby!

Knowing now that this option is rock solid and so quick to implement I am adding this as part of our DR Planning in the future.  My only complaint would be to deploy this solution to all our staff would be fairly cumbersome… but then hopefully the next time anything like this happens we will have AccessACS out of testing and into production.

Maintenance Continues

Our “Extended  Preventative Maintenance has continued to be full of fun and eventful.  The box arrived Wednesday AM from Intel, but the part was a Switch not a Storage Controller.  The modules look very similar and the sku is just on digit difference between the two part codes so one could understand that mistake.  We needed Part # d91231-002 and we received Part # d91241-002.  Needless to say this was very frustrating since I immediately realized this would mean another 24 hours until we received a replacement part.

Jeremie called Intel to alert them of the problem and was told he would receive a call from the RMA-Specialist within the hour to resolve our replacement hardware problem.  For the Next 6 hours every hour we contacted Intel and were told to wait for a call back.  At 4 pm I joined Jeremie on the phone and we were told that the inventory from the parts depot appeared to have the Storage Module on back order. Needless to say steam, fumes, and a few other things started coming out of my ears.

We were talking to a support technician named Carol, who informed me a transfer to her supervisor, a shift manager, a floor manager, or anyone else at Intel was against policy and I would have to wait for the callback. After about 5-10 minutes of this discussion looping Carol informed me that she was ending the call.  ARE YOU KIDDING ME?  A customer calls and informs you that 75 % of the production network is critical down and you say wait for a call back and hang up?  So we immediately called back and talked with another support technician and he again escalated the case.  Within 10 minutes we received a call from Oscar, he was a RMA Specialist and had talked with the staff at the parts depot and confirmed that the Storage Module was on back order… and our option was to purchase a new module from a distributor and Intel would reimburse us.  How can a mission critical part for a system that is 6 months old be on back order???

We called about 10-15 distributors and all were out of stock.  Finally we found our new best friend at SHOPBLT.COM,  Harold returned our voicemail that we left after their stated business hours, and apologized that he didn’t answer the call, but that they were under a tornado warning in Connecticut, but he wanted to know how he could assist us.  He did have one module in a warehouse, and he checked and it was on the shelf and the warehouse manager in Illinois said he would have it on a truck that night.  We asked where in Illinois the warehouse was and could we pick up the part, and Harold informed us he couldn’t disclose the location since they were a defense contractor…

Meanwhile we had contacted our Intel Channel Partner Mark and he had arranged for a test in the Intel Engineering to confirm that the replacement of the module was as simple as putting in the new module.  Intel’s tests confirmed the replacement was as documented.

Thursday AM the part arrived and we reinstalled the original Mid-Plane back into the chassis and said a prayer.  When we powered on the Chassis all was well and we could see and access all our data Yahoo! What a God thing.  Unlike our SAN this hardware was designed correctly to house the information about the RAID arrays on the drives so the volumes were completely intact after we inserted the new storage module.

We copied the data off to our recently made larger SAN and booted up all the virtual servers on various hosts to make sure the data was intact… and it was.  Jeremie then installed the new mid-plane now that the Chassis was not running any virtual servers and upgraded the firmware on the module all with success.  Once MasterFlex was back online we copied the virtual servers back and booted them up… And all production servers back online.

Follow-Up:

- Why don’t we have a second Storage controller in the blade system… $1800 is the cost of the second module.  Would it have prevented this issue.. possibly unless the secondary would have fried during the process to update the firmware too… So now we need to budget for the secondary module.

Questions for Intel:

- Why was there no documentation sent with or online about replacing the Mid-Plane or how to check the firmware version that it would push to all the modules.

- Why is the Storage Module on Back order?  What would Plan C had been if we couldn’t have purchased a module?

- Why is there no escalation process to the Customer Support/Technical Support call queue?

Trip to the City

Natalie and I are taking a trip to the city (Chicago) for our Anniversary.  Its been 6 years already, time flies by when you’re in great company. 

We were able to use a coupon for a free nights stay and a couple rewards points for this trip so we were able to make reservations at the Oak Brook Renaissance Hotel for free!

We ate dinner at Qudoba, we made a deal we can’t eat at any restaurant that we have in Peoria, and Qudoba is one of our favorite fast food Mexican places.

After dinner we headed to Babies R Us (since there isn’t a location in Peoria) to scope out all the baby gear to start to prepare for the arrival of "2.0".  All I have to say is how am I suppose to know how to raise a kid when I can’t even decided which model of 50 strollers we should select…

After my first adventure to Babies R Us, when we came up to our room we found a really awesome treat. The hotel staff had delivered a bottle of sparkling juice and chocolate covered strawberries, a platter of fruit, a note wishing us a happy anniversary and a letter informing us we were the "Guests of the day" with several special privileges including a free on demand movie and appetizers at the restaurant.

We were also informed since I was a member of the rewards program we were receiving complementary breakfast at the restaurant buffet each morning of our stay!  I just love when stuff is FREE!  While the Internet isn’t free, I have my trusty Touch phone with EVDO sharing so we are good.  This hotel recently added some really cool tech, there is a little box near the TV that has RGB, Composite, HDMI and audio inputs which then allows you to  use the LCD tv as your computer monitor for DVD/Video viewing or a really nice size screen to surf on.

After a little surfing it was time for a toast, (Don’t worry Grandmas.. 2.0 is safe, its Sparkling Juice)

Pregnant Lady + Chocolate covered strawberries = Happy Wife

I have to give the staff here at the Renaissance Hotel a big thumbs up for making our first night away a great evening.

Also a thanks to my favorite Platinum Marriott Rewards member for calling the hotel and telling them it was our 6th Anniversary before we arrived! Thanks Dad.

Tomorrow’s plan is a train ride into the City, Lunch, See Wicked and dinner. 

MasterFlex Preventative Maintenance

Every Other Monday night we have our IT Work nights to service our network and Monday night was the night scheduled to replace the mid-plane on MasterFlex.  We had worked with support to identify the cause of the random alerts MasterFlex was sending out and the engineers decided it was the sensors on the Mid-Plane.

So we powered down all the virtual servers and the hosts and took basically everything out of the chassis to prepare for the hardware swap.

The installation was fairly simple after we removed the 12 screws, installed the new mid-plane and installed the modules back in the chassis.

We restarted the chassis to find that the Mid-Plane was a older version of firmware and that the control module was down grading all the components to a previous version… then all went to chaos.

The StorageModule asked to enter safe mode to change the firmware (a process we had done before) but after the module cycled down it never came back online in the management console.

After several hours of tech support calls we identified the module wasn’t powering on, so a part was dispatched.  We continued to talk with support for a little while longer trying to see if there was any resetting or anything of the module that could be done to have the part cycle back on but there was no success.

Around 2 am we were told that the case was escalated to engineering and we would need to wait until engineering contacted us.

For the next 3 1/2 hours Jeremie and I worked to move around virtual servers and recover a few backups to bring online the majority of our services.

When we went home at 5:45 everything (mission critical) but ACS and our Print server were online.  We didn’t bring ACS back from backup because we would loose all of the contributions and other Monday AM processing that had happened after the previous nights backup and prior to our Monday night backup that had not yet happend.. and the Print server didn’t restore from the system state.

So after a few hours of sleep we talk with Intel and they dispatch a new StorageModule /Controller.

More to come as the process continues…..

Church IT RoundTable on UStream

Check out the Central Illinois Church IT RoundTable on Ustream

Symantec AV for Macintosh

Brining online our Mac Server continues….We know that there are very few viruses that are going to harm the OSX machines, but we have still decided install Symantec for Mac on all our OSX machines.  The primary reason is because of the popularity of OSX there is a higher potential for viruses for OSX but also potential for the Macs to be the onramp for harmful files to our network to cause harm to the Windows devices.We downloaded the latest version of Symantec for Mac and the Symantec Admin Console for Mac, which is version 10.2.  The installation process was less than smooth, but that wasn’t to the fault of Apple.  Symantec’s Administration installation guild has a lot to be desired.  Here are some notes from the install.We used the Symantec AntiVirus™ 10 for Macintosh® Installation Guide and the downloaded content from the licensing.Symantec.com web site.The installs for both the Client and Server portions of the software package are fairly straight forward except the guide is not correct or omits valuable information in several areas (noted with “omitted in guide”), your mileage may very but here is the process we used.

• Download the .dmg from licensing.Symantec.com
• Extract the .dmg to the local drive.
• Install MySQL
  • When installing the Console on a Leopard 10.5.3 Server the instructions state that MySQL should be running by default, it isn’t and from what I have read this was a change from 10.4 to 10.5.  The crazy thing is when MySQL isn’t running the installer proceeds and says completed successfully even when it hasn’t.

To enable and configure MySQL to to Server Admin and add the MySQL service.  Next you will need to assign the root login a password.  The default is blank, yet Symantec will not work with a blank password.  To change the password go to terminal and run the following command mysqladmin -u root -h localhost password “newpassword” (replacing newpassword with your selection).

• Enable php on the local web server
  • You next need to check that php is enabled.  The guide makes no mention of needing to use php but after the console install is complete it takes you to a php page, and well by default the OSX 10.5.3 web server does not have the php module enabled.  You will first need to start the Web Service and then enable php5_module.

To enable this go to Server Admin>Servers>OSX Server>Web. Choose the Settings button and the Modules Tab and scroll down to php5_module and check the ‘enable’ check box.

• Assign a static IP address to your OSX server if you haven’t already.
• Run the Symantec Administration Console Installer
  • enter  your admin credentials
  • Name the MySQL database - the default SACM works great
  • Enter the MySQL username: root and the password you set with the command line above.
  • Specify the MySQL database user name the default symadmin works well.
  • enter the credentials you want to use to login to the SAV Console for Mac
  • Choose the Setup Style for the Console, basic works well.
  • enter the host IP address of the OSX server.
  • Enter the Console address and path.  The defaults work well, except the use SSL.  On the first install accessing the Console on port 443 didn’t work but worked on port 80.  It isn’t a major issue in my mind to have this console using SSL so we elected to not use SSL.
  • Enter the Multicast address, the default settings worked well.
  • Create the Key Pairs.
    • Note the Key Pairs will be used to authenticate any command you send from the Console to the clients so choose something here that you will remember and is easy to type.
  • Save the Summary if you would like to document the setup and click Finish.
  • A terminal window will open and the commands will run.
  • Once this is complete you may choose “Open Console” and the console should open if MySQL AND php are running correctly.
• Next you can proceed to install the Client application on those machines you plan to manage with this Console.
• The Installation guide says all you have to do is install the .pkg file found /Library/Application Support/Symantec/SMac/Symantec Administration Client.pkg  this isn’t the whole story. Doing so installs the configuration from your server but doesn’t install the client.  If you are familiar with the windows version of SAV when you push out the client you are doing that pushing out the client configured to check into the server.  This isn’t the case, you must install the client .pkg AND the client application.  Once both are installed and you reboot the machine it should show up in your console.
• After the client and the configuration .pkg are installed you next need to configure the scanning schedule, live updates etc., but you need to know if you are going to do a local LiveUpdate server  to have all your mac clients check into to get the updates rather than having all your machines checking in with Symantec every time they need updates.

(after having done this for as few clients as we have it might not be worth the effort, but its done so i’ll document it.)

• To configure the LiveUpdate server follow these steps (if not using a local update server proceed to the next step in the list)
• View the KB article: How to download and install the LiveUpdate Administration Utility for Macintosh and download the Live Update Admin Install Utility.
  • I wasn’t able to extract the .zip utility on a Mac so i downloaded it to a PC and extracted the .zip and copied it to our server.
• When you install LiveUpdate Administration tool it creates a directory in /applications/liveupdateadminutility.  In this location is the configuration tool and two other directories: Retrieved Updates Retrieved Updates Archives.  These are the default locations for the updates to be stored.
• Since we didn’t want to store the updates on the root volume we created a directory on a second volume called “LiveUpdates” and copied the two directories: Retrieved Updates Retrieved Updates Archives to the new location.
  • Just for others knowledge i created a symbolic link from the original location to the new location just incase someone were to follow the documentation and not know where i saved the updates.
• Next view the KB article: How to configure a Mac OS X Server as an internal LiveUpdate server using HTTP (Web)
  • This KB is really out of order, you first need to decide where you are going to store the updates and note that location.
  • Next go to the directory /library/webserver/documents/  and create a Symbolic Link named LiveUpdate (or what ever subdir path you want to use) pointing to the volume and location where you are saving the updates.
    • Note what you name this Symbolic Link you should know the name of this Symbolic Link is case sensitive in the url for your web server.
    • Brian H @ Symantec suggested a ‘better option’ is to save the updates in the /library/webserver/documents/liveupdate directory but that was on the root volume and we wanted the updates saved on the Storage Volume.
  • In step 4 setting the preferences, when prompted for the location of where you are saving the updates and the expired updates respectively to populate the paths.
    • This can be done by dragging the folders that you created on the Storage volume to the terminal window when prompted for the paths.
• Next View the KB article: How to configure the LiveUpdate Administration Utility for Macintosh
• Using the default settings for each of these properties works well, except for the time of day that you want the LiveUpdate server to download new updates.
• Brian H @ Symantec said that SAV for Mac updates are released only each Friday, but we still choose to check daily at an hour that is in the middle of the night.

• Next you will need to configure the clients to update from the LiveUpdate Server.
• Note that the Host name doesn’t include Http:// and the path is case sensitive.
• After you create the command go to the Send Commands tab and choose send Package.

• Finally you need to tell the clients in what interval to run a scan and live update.  Use the KB Article: How to remotely schedule LiveUpdate and virus scans on Symantec AntiVirus for Macintosh 10.0 clients.
• use the Symsched version commands 4.0.1f1 where “-w 1 23:00 /Users” is 1 the day of the week and 23:00 is the time of day. To set the Scan Interval:
• #!/bin/sh
  #Type your script here
  “/Applications/Symantec Solutions/Symantec Scheduler.app/Contents/Resources/symsched” VirusScan “Weekly Virus Scan” 1 1 -w 1 23:00 /Usersexit 0
• use the Symsched version commands 4.0.1f1 where “-w 4 19:00″ 4 is the day of the week and 19:00 is the time
• #!/bin/sh
  #Type your script here.
  “/Applications/Symantec Solutions/Symantec Scheduler.app/Contents/Resources/symsched” LiveUpdate “Weekly VDefs Update” 1 1 -w 4 19:00 “Virus Definitions” -quietexit 0
• When we ran the scripts we changed the times to be after Friday Night since the tech support told us that most Mac AV updates are released on Fridays of each week.

After these steps are complete your Macs are Running Symantec AV.

Setup of Apple Software Updates

One of our reasons or rather benefits of brining an OSX Open Directory server into the mix was the ability to have a local Mac Update Server.  Granted it hasn’t been a major issue lately since we added a bunch of bandwidth, but we would prefer to have the ability to hold updates until they are tested (at least minimally) before we push them to our client machines.The configuration of Software Update is very simple, a good resource is the apple KB on Software Update Service Overview.A couple “Got Ya’s”

• I started the service and started downloading the updates, but then realized that our test environment didn’t include the two additional volumes our production machine would include.  The two external FireWire drives are for Storage and for TimeMachine.  I stopped the service and created another folder on the root drive then followed the instructions to store Software Update packages on another hard disk or partition. When I re-enabled the service the downloads continued… to the original location.  I stopped the service and deleted the location as mentioned in the KB article and the Update Service didn’t care much for that it just re-created the folders and continued downloading.So before you start the service, run the command to relocate the downloads.In the production installation I installed the service, but before I started the service I followed the instructions to locate the updates on the Storage volume.To configure our location from a terminal window I ran: sudo ln -s /Volumes/Storage/SoftwareUpdates /usr/share/swupd/html

Note in the Administrators guide on page 85 (Chapter 8 Setting Up Software Update Service) the instructions to delete and/or move the updates and create the symbolic link are not correct.

- the guide displays the first command which is correct: 

   sudo rm -rf /usr/share/swupd/html

- the guide displays the second command to move the files: 

mv /usr/share/swupd/html /new_storage_location 

but the command should be 

sudo mv /usr/share/swupd/html /new_storage_location

- the last command in the guide is: 

ln -s /new_storage_location /usr/share/swupd/html

but the command should be: 

 sudo ln -s /new_storage_location /user/share/swupd/html

• To start the download of updates you click the Update List button.  After that it appears that nothing is happening.  Really the list of updates is downloading… and if you enable Automatically copy __ updates from Apple like I did the updates are downloading too.The icon for each update will be gray, once the update is downloaded it turns blue.  Although none of the buttons are grayed out, clicking on them does nothing since well…. its downloading the updates like you said you wanted it to do.  It would be nice though if the buttons were grayed out or there was a status indicator while it was downloading.I let the download go overnight and by morning it was ready to go.To enable individual software updates, select the checkbox in the Enable column of the update.
• We elected to push the settings out to the client OSX machines by Policy in WorkGroup Manager.  One big Got Ya’ when you choose the preference it says to enter the address including /index.suscatalog when you include that the clients error out.  If you simply use http://servername.domain.org:8088 all works well.

IT RoundTable Agenda Taking Shape

I asked those who are planning to attend the Central Illinois Church IT RoundTable on June 12th to begin thinking about some some possible topics, Here are the current suggestions: (in no particular order)

- Security

- Non-Profit pricing

- Computer management

- How do you involve youth in the it ministry?

- How do you keep a grip on technology changes - especially when you have other job responsibilities.

- What local vendors are you using for training / support in IT.

- What about strategy. How did you build it, or is it done on the fly?

- IT Best Practices

- Process documentation

- Policies and Standardization

This list is looking very familiar to the topics popular at the at National RoundTable, and It just further reminds me of the value of this community on every level… across the country or across the street.

I think the thing that I am anticipating the most about this roundtable is the connections with my peers in our local community, but then…some things never change.