Configuring the Control Domain

Now it's time to configure the resources for your control domain! The first step is to make sure that the required SMF services are running:

# svcs -a | grep ldom
online Mar_20 svc:/ldoms/ldmd:default
online Mar_20 svc:/ldoms/vntsd:default

The ldmd service is responsible for controlling the platform and the vntsd service is responsible for providing the virtual terminal services for your logical domains. If they are not running, enable them. You should then be able to run the /opt/SUNWldm/bin/ldm command:

# /opt/SUNWldm/bin/ldm list
Name State Flags Cons VCPU Memory Util Uptime
primary active -t-cv SP 32 32G 0.8% 3d 16h 27m

As you can see, all 32 vcpu's and all of the memory are assigned to the primary (a.k.a. control) domain. We must free up these resources and create the basic infrastructure to support guest domains.

# /opt/SUNWldm/bin/ldm add-vdiskserver primary-vds0 primary
# /opt/SUNWldm/bin/ldm add-vconscon port-range=5000-5100 primary-vcc0 \
primary
# /opt/SUNWldm/bin/ldm add-vswitch net-dev=e1000g0 primary-vsw0 primary
# /opt/SUNWldm/bin/ldm add-vswitch net-dev=e1000g1 primary-vsw1 primary
# /opt/SUNWldm/bin/ldm add-vswitch net-dev=e1000g2 primary-vsw2 primary
# /opt/SUNWldm/bin/ldm add-vswitch net-dev=e1000g3 primary-vsw3 primary
# /opt/SUNWldm/bin/ldm set-mau 1 primary
# /opt/SUNWldm/bin/ldm set-vcpu 4 primary
# /opt/SUNWldm/bin/ldm set-memory 4G primary

The above creates the virtual disk server for servicing storage, the virtual terminal console ports, virtual switch for each physical network port, one crypto unit, 4 vcpu's, and 4GB's of memory for the primary domain. This sets up enough resources for the primary domain, which acts as a control and service domain for the platform. Now we need to store this configuration into the ALOM CMT and reboot.

# /opt/SUNWldm/bin/ldm list-config
factory-default [current]
# /opt/SUNWldm/bin/ldm add-config initial
# /opt/SUNWldm/bin/ldm list-config
factory-default [current]
initial [next]
# shutdown -y -g0 -i6

This stores the configuration and activates it. When the control domain comes back up, you'll notice that the available cpu and memory has changed:

# ldm list primary
Name State Flags Cons VCPU Memory Util Uptime
primary active -t-cv SP 4 4G 0.9% 3d 16h 39m
# psrinfo -vp
The physical processor has 4 virtual processors (0-3)
UltraSPARC-T1 (cpuid 0 clock 1000 MHz)
shou18leng01:~ $ psrinfo -v
Status of virtual processor 0 as of: 04/02/2007 11:00:03
on-line since 03/09/2007 23:53:23.
The sparcv9 processor operates at 1000 MHz,
and has a sparcv9 floating point processor.
Status of virtual processor 1 as of: 04/02/2007 11:00:03
on-line since 03/09/2007 23:53:27.
The sparcv9 processor operates at 1000 MHz,
and has a sparcv9 floating point processor.
Status of virtual processor 2 as of: 04/02/2007 11:00:03
on-line since 03/09/2007 23:53:27.
The sparcv9 processor operates at 1000 MHz,
and has a sparcv9 floating point processor.
Status of virtual processor 3 as of: 04/02/2007 11:00:03
on-line since 03/09/2007 23:53:27.
The sparcv9 processor operates at 1000 MHz,
and has a sparcv9 floating point processor.
# prtdiag -v | grep -i mem
Memory size: 4096 Megabytes

Now we are ready to create our first guest domain! Watch out for the next post.

Disclaimer: This post only covers the Early Access RC3 of the LDOM technology, which is in pre-release.

LDOM Installation

Before you begin, the following is required:

• sun4v based server (SunFire T1000/T2000, Sun Netra T2000, or Sun Netra CP3060 Blade).
• Solaris 10 Update 3 (HW 11/06) or Solaris Express (Build 57 or higher) installed.
• Logical Domains 1.0 Early Access

The first step is into install the firmware included with the LDOM software bundle. The firmware will contain the ALOM CMT, Post, OBP, and hypervisor updates. You must load the corresponding firmware for your platform. There are two methods for doing this. You can download the firmware to the ALOM CMT using FTP or you can upload it from your currently installed Solaris instance. The later is much simpler:)

# cd Firmware/tools # ./sysfwdownload ../Sun_System_Firmware-6_4_0_build_07-Sun_Fire_T2000.bin

.......... (10%).......... (20%).......... (30%).......... (40%).......... (51%)
.......... (61%).......... (71%).......... (81%).......... (92%)........ (100%)

Download completed successfully.

This will upload the firmware to your ALOM CMT. Make sure that you upload the corresponding firmware for your platform. Now you need to shutdown your Solaris instance:

# shutdown -y -g0 -i5

Now you can upgrade the firmware from the ALOM CMT console:

sc> showkeyswitch
Keyswitch is in the NORMAL position.
sc>
SC Alert: Host system has shut down.
flashupdate -s 127.0.0.1

SC Alert: System poweron is disabled.
................................................................................
................................................................................
......

Update complete. Reset device to use new software.

SC Alert: SC firmware was reloaded
sc> resetsc
Are you sure you want to reset the SC [y/n]? y

The firmware is now updated and the SC has been reset. Once it is done resetting, verify the version of the firmware:

sc> showhost
Sun-Fire-T2000 System Firmware 6.4.0_build_07 2007/02/14 22:07

Host flash versions:
Hypervisor 1.4.0_build_07 2007/02/14 21:52
OBP 4.26.0.build_07 2007/02/14 19:20
POST 4.26.0.build_07 2007/02/14 19:51

The version should match the version info in the firmware bin file name. Now you can power on your server and proceed to the installation of the LDOM software. Depending on the OS you are running, you may have to apply the patches that are included in the Patches directory first.

For example, if you are running Solaris 10 Update 3, you will need to install 118833-36 and reboot. Then you'll have to install patches 125043-01 and T124921-02, then reboot. This is not required if you are running build 57 or higher of Nevada (OpenSolaris, Solaris Express, etc.).

Now it's time to install the LDOM software for what will become the control domain. The software package includes JASS to secure the control domain. Remember, the control domain is similar to the SC on a Sun Fire 15K. You don't want it to be used for anything other than administering the platform. You can install the SUNWjass and SUNWldm package with the install-ldm script under the Install directory. Or you can install them manually. If you already have secured the control domain, you may not need JASS, it's up to you:)

# Install/install-ldm
Welcome to the LDoms installer.

You are about to install the domain manager package that will enable
you to create, destroy and control other domains on your system. Given the capabilities of the domain manager, you can now change the security configuration of this Solaris instance using the Solaris Security Toolkit.
Select a security profile from this list:
a) Hardened Solaris configuration for LDoms (recommended)
b) Standard Solaris configuration
c) Your custom-defined Solaris security configuration profile
Enter a, b, or c [a]: a
The changes made by selecting this option can be undone through the
Solaris Security Toolkit’s undo feature. This can be done with the
’/opt/SUNWjass/bin/jass-execute -u’ command.
Installing LDoms and Solaris Security Toolkit packages.

Installation of was successful.
...
Verifying that all packages are fully installed. OK.
Enabling services: svc:/ldoms/ldmd:default
Running Solaris Security Toolkit 4.2.0 driver ldm_control-secure.driver.
...
Solaris Security Toolkit hardening executed successfully; log file
/var/opt/SUNWjass/run//jass-install-log.txt. It will not
take effect until the next reboot. Before rebooting, make sure SSH or
the serial line is setup for use after the reboot.

Then reboot your control domain. Once you come backup. It's time to start setting things up! We'll visit that in my next post!

Disclaimer: This post only covers the Early Access RC3 of the LDOM technology, which is in pre-release.

Introduction to LDOM's

Logical domains are discrete instances of the Solaris OE running independently within a virtualized environment. Each logical domain has its own virtual cpu, memory, OBP, console, networking, storage, and I/O components. These components are configured with a combination of different technologies.

• sun4v Platform Hypervisor
• Logical Domain Management Software
• Solaris OE

The hypervisor provides the mechanism for masking and virtualizing the resources on the platform. The hypervisor is a light-weight software layer built into the ALOM CMT firmware. The hypervisor also helps to abstract the low-level hardware details from logical domains.

The logical domain management software is the nexus for control and configuration of the hypervisor. This software provides a CLI to controlling and configuring the resources that define each logical domain. Only one logical domain can run the management software. This logical domain is known as the "primary" or control domain. More about the different LDOM types in a moment.

The Solaris OE provides support for the sun4v platform, dynamic reconfiguration, and virtual devices. At this time, you need Solaris 10 Update 3 (11/06) or Nevada build 57. It's not possible to use Solaris 9 or below for LDOM's, as the platform support is not there.

There are four types of LDOM's that can be created:

• Control Domain
• Service Domain
• I/O Domain
• Guest Domain

The control domain is the first installed LDOM or instance of Solaris on the platform. This LDOM contains the Logical Domain Management (SUNWldm) software for managing the platform. It is from this LDOM that all of the hardware platform specifics are visible and configurable. The control and configuration of the platform is communicated through LDC's (Logical Domain Channels). It is through this mechanism that the configuration, virtual devices, and virtual services communications are relayed.

A service domain is an LDOM that has control over either one or more PCI-E controllers. It consists of an instance of the Solaris OE. No additional software is required, the control domain will configure the virtualized devices and service within a service domain. The service domain will then service the I/O for these virtualized components for guest domains to utilize. The service domain has direct control over the hardware under its PCI-E controller. There are only two in the Sun Fire T2000. So only two service domains are configurable, one of which must also be the control domain. It is important to remember that the control domain is one of the service domains. If a second service domain is created, this is called a Split PCI-E configuration. More about that later.

An I/O domain is exactly like a service domain, except for the fact that none of its devices or services are virtualized for guest domains. This is useful if you have an application that requires direct access to a PCI-E device for performance or some other reason.

A guest domain is a consumer of virtualized devices and services. Meaning that it does not virtualize any devices or services for other domains. It is independent of other guest domains. However, it is dependent upon the service domains that provide its virtual devices and services. A guest domain consist of its own instance of Solaris OE. This is where your applications will typically live as consuming resources in the control or services domains affects the platform as a whole.

While a fully configured Sun Fire T2000 has a total of 32 CPU threads, it's probably not a good idea to create 32 LDOM's. As this would under power the control and services domains.

The next post will be about the installation of the firmware, patches, and Logical Domain Management software.

Disclaimer: This post only covers the Early Access RC3 of the LDOM technology, which is in pre-release.

LDOM Posts are a go!

Well fantastic news arrived today from Sun. The licensing agreement for the LDOM Early Access download has been changed to allow blogging and the writing of articles. Big thanks to Sun and the LDOM team! So keep an eye on my blog, I'll start posting entries about my experiences with LDOM's this weekend.

Some changes in Solaris Express Build 59

I've been running build 59 on my laptop since last week. So far my experiences have been good. I'm getting ready to do the jumpstart shuffle and migrate my Sun Blade 2000 to build 59. I have noticed a few changes:

• Developer Edition Installer (Requires over 700MB's RAM to start)
• Developer tools included on DVD media.
  
• /usr/gnu popped up! (Currently only GNU m4 is in there..)
• /usr/postgres popped up!
• SunVTS is now part of Solaris under /usr/sunvts!
• Lots of resource enhancements for zones!
  
• Firefox works with Yahoo's beta email again!

I'm really glad to see the integration of the Sun Studio tools with OpenSolaris. I'm hoping that the same happens for the SPARC release as well (Don't want to leave ppl like me out!). It's great to see the freeware tools move from /usr/bin and /usr/sfw/bin into their own directory structure. I thought it was a big mistake to dump everything into /usr/bin. I'm looking forward to these kinds of clean ups and hopefully the removal of /usr/sfw altogether.

The zone enhancements are amazing (swap sets, memory capping, etc.). I'll have to update my mods to the JET zone module. This time, I'm going to publicly post my changes. But first, I have to test them out with my Sun Blade 2000 and make sure they work!

Sorry for the delay in the JET documentation, it's coming I assure you!

Update on LDOM posts..

I'm working with Sun to clear up licensing issues with posting about LDOM's. Hopefully, I'll have this resolved shortly and be able to proceed with posting. Stay tuned:)

LDOM Posts will have to wait :(

Well it's a good thing I asked first! Turns out that I'll have to wait a little while longer before I can blog about LDOM's publicly. Sorry for the confusion, but I'll have to ask everyone to wait a little longer. In the meantime, I'm going to get my notes together to write a few articles for release once LDOM's have gone GA (General Access). Sorry again!

Solaris Express Community Edition Build 59 is out!

Build 59 is out. Check out the OpenSolaris links to the far right to download it today. This weekend I'll be doing the jumpstart shuffle to upgrade. I really want to test out the resource enhancements for zones.

I posted my changes to the JetZONE module to the JET team, hopefully they'll make use of my changes. I'll post some more info on the changes and see if I can find somewhere to host them. I've been working on a ZFS module for JET, so it'll be a good test to see how well it works!

Logical Domains 1.0 Early Access Available!

Well it's about time! I've been a beta tester for LDom's for a while now. Finally, I can talk about it! Over the next few days, I'll be making lots of posts about my experiences with LDom's and give quick how-to's. In the mean time, here are the relevant links to get started:

• Logical Domain 1.0 Early Access
• Beginners Guide to LDoms: Understanding and Deploying Logical Domains

Enjoy!