The Problem: We ran out of IPs on an internal VLAN used for development environments. I wanted a way to utilze both the old and the new vlans on multiple openvz servers with out losing any of the extra bandwidth provided by bonding. Furthermore I did not want to lose the ability to move VE's between different Hardware nodes.

Update 20110819: Per ian@ianbmacdonald.com comment "The parameters need to be bond_xmit_hash_policy and bond_lacp_rate. ... You can see in the /proc/net/bonding/bond0 output that the policy is set to "layer2" not "layer 2+3" as per the configuration (because of this error)." I have updated the /etc/init.d/interfaces entry below to reflect this.

The Solution:
Setup a new Debian Squeeze Openvz server with bonding (802.3ad) and vlan turnking(802.1Q). This article covers the process of getting vlan and bonding working on Debian Squeeze with a cisco switch running IOS.

Cisco Setup:
Cisco Hardware
We are using a cisco 6509 switch with gigabit ethernet module that supports 802.3ad. For more information regarding the different bonding options you can check out this link I have not tried getting this to work with non 802.3ad (Dynamic link aggregation) capable switch.

Setup the port channel

interface Port-channel30
 description ServerName
 switchport
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 48,49
 switchport mode trunk
 no ip address
end

Configure the physical interfaces on the cisco switch:

interface GigabitEthernet9/5
 description ServerName#1
 switchport
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 48,49
 switchport mode trunk
 no ip address
 stack-mib portname ServerName#1
 no snmp trap link-status
 no cdp enable
 channel-protocol lacp
 channel-group 30 mode active
end

interface GigabitEthernet9/19
 description ServerName#2
 switchport
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 48,49
 switchport mode trunk
 no ip address
 stack-mib portname ServerName#2
 no snmp trap link-status
 no cdp enable
 channel-protocol lacp
 channel-group 30 mode active
end
...

Make sure the the "switchport trunk allowed vlan" has the vlans you are going to be doing on the linux server. Until these matched it would not work for me.

Linux Network Config:
Install the required pacakges and load bonding module

apt-get install vlan ifenslave
modprobe bonding

Interfaces Config: /etc/network/interfaces

auto bond0
iface bond0 inet manual
        bond-mode 4
        bond-miimon 100
        bond_xmit_hash_policy layer2+3
        bond_lacp_rate slow
        slaves eth0 eth1 eth2 eth3

auto vlan48
iface vlan41 inet static
        vlan_raw_device bond0
        address 10.169.48.77
        netmask 255.255.255.0
        network 10.169.48.0
        broadcast 10.169.48.255
        gateway 10.169.48.1

auto vlan49
iface vlan49 inet static
        vlan_raw_device bond0
        address 10.169.49.45
        netmask 255.255.255.0
        network 10.169.49.0
        broadcast 10.169.49.255
        gateway 10.169.49.1

If you happen to be using openvz I set the below for /etc/sysctl.conf. I have removed all of the comments and blank lines. You do not need this if you are not using OpenVZ.

egrep -v '^#|^$' /etc/sysctl.conf 
net.ipv4.icmp_echo_ignore_broadcasts=1
net.ipv4.conf.eth0.proxy_arp=1
net.ipv4.conf.bond0.proxy_arp=1
net.ipv4.conf.default.forwarding=1 
net.ipv4.conf.default.proxy_arp = 0
net.ipv4.ip_forward=1 
net.ipv4.conf.all.rp_filter = 0
kernel.sysrq = 1
net.ipv4.conf.default.send_redirects = 1
net.ipv4.conf.all.send_redirects = 0
fs.file-max = 100000

sysctl is used on bootup so you need to run the below command to load the file.

/sbin/sysctl -p

Trouble Shooting:
On Linux

ServerName# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.5.0 (November 4, 2008)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

802.3ad info
LACP rate: slow
Aggregator selection policy (ad_select): stable
Active Aggregator Info:
        Aggregator ID: 7
        Number of ports: 4
        Actor Key: 17
        Partner Key: 30
        Partner Mac Address: 00:15:2c:79:c4:c0

Slave Interface: eth0
MII Status: up
Link Failure Count: 0
Permanent HW addr: d4:85:64:54:1d:5c
Aggregator ID: 7

Slave Interface: eth1
MII Status: up
Link Failure Count: 0
Permanent HW addr: d4:85:64:54:1d:5e
Aggregator ID: 7

Slave Interface: eth2
MII Status: up
Link Failure Count: 1
Permanent HW addr: d4:85:64:54:1d:84
Aggregator ID: 7

Slave Interface: eth3
MII Status: up
Link Failure Count: 1
Permanent HW addr: d4:85:64:54:1d:86
Aggregator ID: 7

ServerName# modinfo bonding 
filename:       /lib/modules/2.6.32-5-openvz-amd64/kernel/drivers/net/bonding/bonding.ko
author:         Thomas Davis, tadavis@lbl.gov and many others
description:    Ethernet Channel Bonding Driver, v3.5.0
version:        3.5.0
license:        GPL
srcversion:     C0EFCD8CB4AC214A8146EC2
depends:        
vermagic:       2.6.32-5-openvz-amd64 SMP mod_unload modversions 
parm:           max_bonds:Max number of bonded devices (int)
parm:           num_grat_arp:Number of gratuitous ARP packets to send on failover event (int)
parm:           num_unsol_na:Number of unsolicited IPv6 Neighbor Advertisements packets to send on failover event (int)
parm:           miimon:Link check interval in milliseconds (int)
parm:           updelay:Delay before considering link up, in milliseconds (int)
parm:           downdelay:Delay before considering link down, in milliseconds (int)
parm:           use_carrier:Use netif_carrier_ok (vs MII ioctls) in miimon; 0 for off, 1 for on (default) (int)
parm:           mode:Mode of operation : 0 for balance-rr, 1 for active-backup, 2 for balance-xor, 3 for broadcast, 4
for 802.3ad, 5 for balance-tlb, 6 for balance-alb (charp)
parm:           primary:Primary network device to use (charp)
parm:           lacp_rate:LACPDU tx rate to request from 802.3ad partner (slow/fast) (charp)
parm:           ad_select:803.ad aggregation selection logic: stable (0, default), bandwidth (1), count (2) (charp)
parm:           xmit_hash_policy:XOR hashing method: 0 for layer 2 (default), 1 for layer 3+4 (charp)
parm:           arp_interval:arp interval in milliseconds (int)
parm:           arp_ip_target:arp targets in n.n.n.n form (array of charp)
parm:           arp_validate:validate src/dst of ARP probes: none (default), active, backup or all (charp)
parm:           fail_over_mac:For active-backup, do not set all slaves to the same MAC.  none (default), active or
follow (charp)

On Cisco

show interfaces port-channel 30
Port-channel30 is up, line protocol is up (connected)
  Hardware is EtherChannel, address is 0013.80c0.fa4c (bia 0013.80c0.fa4c)
  Description: Punkinpuss
  MTU 1500 bytes, BW 4000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s
  input flow-control is off, output flow-control is off
  Members in this channel: Gi9/5 Gi9/19 Gi11/45 Gi12/45 
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output never, output hang never
  Last clearing of "show interface" counters 4w3d
  Input queue: 0/2000/7/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 34000 bits/sec, 8 packets/sec
  5 minute output rate 120000 bits/sec, 112 packets/sec
     13303252 packets input, 1748466512 bytes, 0 no buffer
     Received 103127 broadcasts (101124 multicasts)
     2 runts, 0 giants, 0 throttles
     5 input errors, 0 CRC, 0 frame, 2 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     111206034 packets output, 42975015356 bytes, 0 underruns
     3 output errors, 0 collisions, 1 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

Links:
openvz on debian
ubnutu bug report where I found my answer
bondong on debian
bondong on debian in a vmware instance

Conclusion:
I had a hard time finding all of the information required to setup vlan and bonding under squeeze so I put this howto together. Please feel free to post any questions or comments.